KeepLearningBigData

基因数据处理52之cs-bwamem集群版运行（1千万条100bp的reads）

1.art生成模拟序列：

 art_illumina -ss HS20 -i GRCH38BWAindex/GRCH38chr1L3556522.fna -l 100 -c 10000000 -o g38L100c10000000Nhs20

2.上传到hdfs，制定partition数

spark-submit  --class cs.ucla.edu.bwaspark.BWAMEMSpark --master spark://masterIP:7077  /home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar upload-fastq   0 21 g38L100c10000000Nhs20.fq /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.fastq

masterIP为真实IP

3.使用cs-bwamem进行匹配

 spark-submit --executor-memory 4g --class cs.ucla.edu.bwaspark.BWAMEMSpark --total-executor-cores 20 --master spark://masterIP:7077  --conf spark.driver.host=masterIP --conf spark.driver.cores=4 --conf spark.driver.maxResultSize=4g --conf spark.storage.memoryFraction=0.7  --conf spark.akka.threads=2 --conf spark.akka.frameSize=1024 /home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar cs-bwamem -bfn 1 -bPSW 1 -sbatch 10 -bPSWJNI 1  -oChoice 2 -oPath hdfs://masterIP:9000/xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam -localRef 1  -isSWExtBatched 1  0 GRCH38BWAindex/GRCH38chr1L3556522.fasta  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.fastq

4.将数据进行合并

spark-submit --executor-memory 4g --class cs.ucla.edu.bwaspark.BWAMEMSpark --total-executor-cores 4 --master spark://masterIP:7077  --conf spark.driver.host=masterIP --conf spark.driver.cores=4 --conf spark.driver.maxResultSize=6g --conf spark.storage.memoryFraction=0.7  --conf spark.akka.threads=2 --conf spark.akka.frameSize=1024 /home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jar merge hdfs://masterIP:9000 /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam

5.统计数量
行数：34253515
distinct:34253515

分析：distinct显示没有重复的数据
问题：distinct耗时太长，本地得30分钟

参考：

https://github.com/ytchen0323/cloud-scale-bwamem/blob/453a4178cd6c3fb3a0fe99df816e6cd7df521b95/src/main/scala/cs/ucla/edu/bwaspark/Usage.scala

附录：
（1）部分df。show：

+--------------------+---------+---------+----+------------+--------------------+--------------------+-----+---------------------+-------------------+----------+----------+----------+----------+-----------+------------+-------------------------+-------------+------------------+------------------+----------------+------------------+----------------------+--------------------+--------+--------------------+---------------+---------------------------+----------------------+-----------------------+--------------------+----------------------+------------------+------------------------------------+-------------------+-----------------------+-----------------+------------------+----------------+----------+
|              contig|    start|      end|mapq|    readName|            sequence|                qual|cigar|basesTrimmedFromStart|basesTrimmedFromEnd|readPaired|properPair|readMapped|mateMapped|firstOfPair|secondOfPair|failedVendorQualityChecks|duplicateRead|readNegativeStrand|mateNegativeStrand|primaryAlignment|secondaryAlignment|supplementaryAlignment|mismatchingPositions|origQual|          attributes|recordGroupName|recordGroupSequencingCenter|recordGroupDescription|recordGroupRunDateEpoch|recordGroupFlowOrder|recordGroupKeySequence|recordGroupLibrary|recordGroupPredictedMedianInsertSize|recordGroupPlatform|recordGroupPlatformUnit|recordGroupSample|mateAlignmentStart|mateAlignmentEnd|mateContig|
+--------------------+---------+---------+----+------------+--------------------+--------------------+-----+---------------------+-------------------+----------+----------+----------+----------+-----------+------------+-------------------------+-------------+------------------+------------------+----------------+------------------+----------------------+--------------------+--------+--------------------+---------------+---------------------------+----------------------+-----------------------+--------------------+----------------------+------------------+------------------------------------+-------------------+-----------------------+-----------------+------------------+----------------+----------+
|[chr1,248956422,n...| 14809719| 14809819|  60|chr1-6811129|CATGTACAGTGCTCGCC...|CEDA8DA@E<DDED.DE...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|            true|             false|                 false|            34T46C18|    null|NM:i:2    AS:i:90 XS...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 80256345| 80256445|  60|chr1-6811128|AGTAGGCTTGGAGAAGA...|D?BCED>D=@DDEC9CF...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|            true|             false|                 false|                 100|    null|NM:i:0    AS:i:100    X...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|173354132|173354232|  60|chr1-6811126|AGCACAATCAGAAATAA...|?@@FFDFF2?HHHJ3JJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|            true|             false|                 false|               29T70|    null|NM:i:1    AS:i:95 XS...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...| 67191506| 67191606|  60|chr1-6811125|CATTCCCATTGGCTTTT...|?C@F=DEFHHHHHFHIH...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|            true|             false|                 false|                 100|    null|NM:i:0    AS:i:100    X...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143209655|143209755|   5|chr1-6811124|ATTCCATTTGATGACAA...|D5>D:DBCCAC5>,(AC...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|            true|             false|                 false|            31A44C23|    null|NM:i:2    AS:i:90 XS...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|125183587|125183687|   0|chr1-6811124|ATTCCATTTGATGACAA...|D5>D:DBCCAC5>,(AC...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|          31A0A43C23|    null|NM:i:3    AS:i:85 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|125166941|125167041|   0|chr1-6811124|ATTCCATTTGATGACAA...|D5>D:DBCCAC5>,(AC...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|          31A0A43C23|    null|NM:i:3    AS:i:85 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|125181832|125181932|   0|chr1-6811124|ATTCCATTTGATGACAA...|D5>D:DBCCAC5>,(AC...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|          31A0A43C23|    null|NM:i:3    AS:i:85 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143271137|143271237|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|          23G43T0T31|    null|NM:i:3    AS:i:85 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143211045|143211145|   0|chr1-6811124|ATTCCATTTGATGACAA...|D5>D:DBCCAC5>,(AC...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|        27A3A0A43C23|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143189224|143189324|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|       23G22G20T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143233259|143233359|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|        6T16G43T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143274817|143274917|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|       23G14T28T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143215364|143215464|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|        6T16G43T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|125180499|125180599|   0|chr1-6811124|ATTCCATTTGATGACAA...|D5>D:DBCCAC5>,(AC...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|        31A0A43C16A6|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143267274|143267374|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|        6T16G43T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143226169|143226269|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|       23G14T28T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143264022|143264122|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|        6T16G43T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143223869|143223969|   0|chr1-6811124|TGGAATCATGATCAAAT...|C@CFFFFDGH:HAJJIJ...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|             false|             false|           false|              true|                 false|       23G25C17T0T31|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
|[chr1,248956422,n...|143258530|143258630|   0|chr1-6811124|ATTCCATTTGATGACAA...|D5>D:DBCCAC5>,(AC...| 100M|                    0|                  0|     false|     false|      true|     false|      false|       false|                    false|        false|              true|             false|           false|              true|                 false|       31A0A17G25C23|    null|NM:i:4    AS:i:80 RG...|            foo|                       null|                  null|                   null|                null|                  null|              null|                                null|               null|                   null|              bar|              null|            null|      null|
+--------------------+---------+---------+----+------------+--------------------+--------------------+-----+---------------------+-------------------+----------+----------+----------+----------+-----------+------------+-------------------------+-------------+------------------+------------------+----------------+------------------+----------------------+--------------------+--------+--------------------+---------------+---------------------------+----------------------+-----------------------+--------------------+----------------------+------------------+------------------------------------+-------------------+-----------------------+-----------------+------------------+----------------+----------+
only showing top 20 rows

（2）合并前后数据展示：

hadoop@Master:~$ hadoop fs -du -h /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/0
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/1
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/10
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/100
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/101
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/102
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/103
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/104
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/105
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/106
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/107
9.6 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/108
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/109
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/11
9.7 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/110
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/111
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/112
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/113
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/114
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/115
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/116
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/117
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/118
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/119
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/12
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/120
10.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/121
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/122
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/123
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/124
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/125
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/126
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/127
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/128
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/129
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/13
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/130
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/131
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/132
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/133
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/134
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/135
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/136
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/137
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/138
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/139
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/14
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/140
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/141
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/142
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/143
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/144
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/145
10.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/146
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/147
1.6 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/148
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/15
9.7 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/16
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/17
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/18
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/19
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/2
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/20
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/21
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/22
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/23
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/24
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/25
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/26
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/27
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/28
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/29
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/3
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/30
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/31
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/32
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/33
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/34
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/35
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/36
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/37
10.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/38
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/39
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/4
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/40
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/41
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/42
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/43
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/44
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/45
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/46
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/47
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/48
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/49
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/5
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/50
10.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/51
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/52
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/53
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/54
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/55
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/56
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/57
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/58
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/59
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/6
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/60
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/61
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/62
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/63
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/64
9.7 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/65
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/66
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/67
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/68
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/69
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/7
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/70
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/71
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/72
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/73
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/74
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/75
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/76
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/77
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/78
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/79
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/8
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/80
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/81
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/82
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/83
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/84
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/85
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/86
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/87
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/88
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/89
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/9
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/90
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/91
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/92
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/93
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/94
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/95
10.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/96
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/97
9.9 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/98
9.8 M   /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.adam/99
hadoop@Master:~$ hadoop fs -du -h /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam
du: `/xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam': No such file or directory
hadoop@Master:~$ hadoop fs -du -h /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam
0      /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/_SUCCESS
9.7 K  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/_common_metadata
1.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/_metadata
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00000.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00001.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00002.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00003.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00004.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00005.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00006.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00007.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00008.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00009.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00010.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00011.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00012.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00013.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00014.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00015.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00016.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00017.gz.parquet
4.3 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00018.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00019.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00020.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00021.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00022.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00023.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00024.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00025.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00026.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00027.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00028.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00029.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00030.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00031.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00032.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00033.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00034.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00035.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00036.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00037.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00038.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00039.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00040.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00041.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00042.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00043.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00044.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00045.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00046.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00047.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00048.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00049.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00050.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00051.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00052.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00053.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00054.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00055.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00056.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00057.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00058.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00059.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00060.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00061.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00062.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00063.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00064.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00065.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00066.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00067.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00068.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00069.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00070.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00071.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00072.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00073.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00074.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00075.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00076.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00077.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00078.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00079.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00080.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00081.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00082.gz.parquet
3.8 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00083.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00084.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00085.gz.parquet
3.8 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00086.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00087.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00088.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00089.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00090.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00091.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00092.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00093.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00094.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00095.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00096.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00097.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00098.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00099.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00100.gz.parquet
3.8 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00101.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00102.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00103.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00104.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00105.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00106.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00107.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00108.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00109.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00110.gz.parquet
3.8 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00111.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00112.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00113.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00114.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00115.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00116.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00117.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00118.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00119.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00120.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00121.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00122.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00123.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00124.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00125.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00126.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00127.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00128.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00129.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00130.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00131.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00132.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00133.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00134.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00135.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00136.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00137.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00138.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00139.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00140.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00141.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00142.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00143.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00144.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00145.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00146.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00147.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00148.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00149.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00150.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00151.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00152.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00153.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00154.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00155.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00156.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00157.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00158.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00159.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00160.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00161.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00162.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00163.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00164.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00165.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00166.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00167.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00168.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00169.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00170.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00171.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00172.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00173.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00174.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00175.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00176.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00177.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00178.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00179.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00180.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00181.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00182.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00183.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00184.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00185.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00186.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00187.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00188.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00189.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00190.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00191.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00192.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00193.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00194.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00195.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00196.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00197.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00198.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00199.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00200.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00201.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00202.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00203.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00204.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00205.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00206.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00207.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00208.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00209.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00210.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00211.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00212.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00213.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00214.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00215.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00216.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00217.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00218.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00219.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00220.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00221.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00222.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00223.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00224.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00225.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00226.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00227.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00228.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00229.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00230.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00231.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00232.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00233.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00234.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00235.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00236.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00237.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00238.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00239.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00240.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00241.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00242.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00243.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00244.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00245.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00246.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00247.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00248.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00249.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00250.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00251.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00252.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00253.gz.parquet
4.3 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00254.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00255.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00256.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00257.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00258.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00259.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00260.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00261.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00262.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00263.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00264.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00265.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00266.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00267.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00268.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00269.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00270.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00271.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00272.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00273.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00274.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00275.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00276.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00277.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00278.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00279.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00280.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00281.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00282.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00283.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00284.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00285.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00286.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00287.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00288.gz.parquet
4.1 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00289.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00290.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00291.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00292.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00293.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00294.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00295.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00296.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00297.gz.parquet
4.2 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00298.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00299.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00300.gz.parquet
4.0 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00301.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00302.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00303.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00304.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00305.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00306.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00307.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00308.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00309.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00310.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00311.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00312.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00313.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00314.gz.parquet
3.5 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00315.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00316.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00317.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00318.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00319.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00320.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00321.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00322.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00323.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00324.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00325.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00326.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00327.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00328.gz.parquet
3.8 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00329.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00330.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00331.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00332.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00333.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00334.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00335.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00336.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00337.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00338.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00339.gz.parquet
3.8 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00340.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00341.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00342.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00343.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00344.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00345.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00346.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00347.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00348.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00349.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00350.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00351.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00352.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00353.gz.parquet
3.6 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00354.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00355.gz.parquet
3.7 M  /xubo/data/alignment/cs-bwamem/fastq/g38L100c10000000Nhs20.merge.adam/part-r-00356.gz.parquet
hadoop@Master:~$

你可能感兴趣的:(基因数据处理)

R语言零基础基因/数据差异分析（一） Frms R语言零基础基因/数据差异分析 r语言数据分析数据可视化
文章目录介绍环境搭建软件下载结果展示基因数据下载流程基因数据处理利用GEO分析绘制拟火山图注意，本系列有连贯性，每一步都很详细，每一步都很重要，请耐心读完！！介绍本系列文主要依据真实论文制图流程，详细说明制图过程，其中包括：1.基因数据下载2.制图所需数据格式3.火山图制作流程4.聚类热图制作流程环境搭建软件下载移步至此学习结果展示基因数据处理注意删除末行注释基因数据下载流程以GSE137578基
PyVCF 变异基因数据处理 loong_XL 生信生信
PyVCF是一个用于处理VCF(VariantCallFormat)文件的python库。它提供了许多功能来读取，过滤和修改VCF文件中的变异PyVCF是一个用于读取和写入VCF格式文件的Python库。主要功能包括:1、读取VCF文件：PyVCF提供了一个vcf.Reader()函数，可以用来打开并读取VCF文件。读取后的文件可以进行遍历，每个元素是一个vcf.model._Record的实例2
云上弹性高性能计算，支持生命科学产业高速发展、降本增效阿里云开发者人工智能运维数据可视化安全 BI 数据处理调度云计算数据中心容器阿里云开发者
简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整个产业的转型升级，
云上弹性高性能计算，支持生命科学产业高速发展、降本增效程序员
简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？image.png随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整
云上弹性高性能计算，支持生命科学产业高速发展、降本增效人工智能运维数据可视化安全bi
简介：生命科学的研究早已离不开高性能计算（HighPerformanceComputing，HPC）的辅助。从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。而云，又能如何助力产业发展，帮助产业降本增效呢？随着云计算技术服务及实践的日趋成熟，越来越多的行业通过上云实现了整个产业的转型升级，
相似度融合网络：用于聚合不同的基因数据类型：Similarity network fusion for aggregating data types on a genomic scale Ice-iron 论文笔记机器学习神经网络深度学习
论文标题：Similaritynetworkfusionforaggregatingdatatypesonagenomicscale.论文下载地址论文总结论文以计算机视觉多视图方式为启发，设计了一种图融合网络用于解决基因数据不能综合处理的困难。现有的基因数据非常丰富，有各种类型的基因数据可以利用。但现有的基因数据处理方式大多数是只利用一种基因数据，例如只使用DNA或者是只使用mRNA，不能综合所有
基因数据处理80之disease的DataProcessing KeepLearningBigData 基因数据处理
1.代码：/***@authorxubo*morecode:https://github.com/xubo245/SparkLearning*moreblog:http://blog.csdn.net/xubo245*/packageorg.gcdss.cli.diseaseimportjava.text.SimpleDateFormatimportjava.util.Dateimportorg.
基因数据处理85之adam-0.18.2无法读取0.14.0使用adamSave存储的数据 KeepLearningBigData 基因数据处理
1.介绍：cs-bwamem依赖的是adam-0.14.0里面的adamSave存储之后无法用adam-0.18.2的结果读取：2.adam-0.18.2adamsave可以用loadParquetAlignments读取。但无法用loadParquetAlignments读取adam-0.14.23.代码：packageorg.gcdss.cli.testimportjava.nio.file.
基因数据处理73之从HDFS读取fasta文件存为Adam的parquet文件 KeepLearningBigData 基因数据处理
1.GRCH38chr14：hadoop@Master:~/xubo/project/load$./load.shstart:1SLF4J:Failedtoloadclass"org.slf4j.impl.StaticLoggerBinder".SLF4J:Defaultingtono-operation(NOP)loggerimplementationSLF4J:Seehttp://www.sl
基因数据处理94之使用kmer分析SRR003161数据的kmer分布 KeepLearningBigData 基因数据处理
1.分两组(1)kmer长度为：5to21(2)kmer长度为：5to55by102.代码：packageorg.gcdss.cliimportjava.text.SimpleDateFormatimportjava.util._importorg.apache.spark._importorg.bdgenomics.adam.projections.{AlignmentRecordField,P
基因数据处理57之BWA-MEM运行single-end(1千万条100bp的reads) KeepLearningBigData 基因数据处理
```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20.fq>g38L100c10000000Nhs20.bwamem.sam[M::bwa_idx_load_from_disk]read0ALTcon
基因数据处理75之从HDFS读取vcf文件存为Adam的parquet文件（成功） KeepLearningBigData 基因数据处理
1.参考：packageorg.bdgenomics.adam.cliclassFlattenSuiteextendsADAMFunSuite{valloader=Thread.currentThread().getContextClassLoadervalinputPath=loader.getResource("small.vcf").getPathvaloutputFile=File.cre
基因数据处理74之从HDFS读取vcf文件存为Adam的parquet文件（有问题） KeepLearningBigData 基因数据处理 spark 基因数据处理 adam
1.small.vcf:没记录2.读取：5loadtime:3287ms{"variant":{"variantErrorProbability":139,"contig":{"contigName":"1","contigLength":null,"contigMD5":null,"referenceURL":null,"assembly":null,"species":null,"refere
基因数据处理2之ftp数据快速查找 KeepLearningBigData 基因数据处理
基因数据处理2之ftp数据快速查找linux下可以用grep:curl-s"ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/current.tree"|grepNA12878运行结果：hadoop@Mcnode1:~/cloud/adam/xubo/testAdam34/TestBaiBas$curl-s"ftp://ftp.1000genomes.ebi.ac.
基因数据处理83之移动GRCH38Index到每个节点 KeepLearningBigData 基因数据处理
1.从cloud/adam移出到xubo/ref:hadoop@Master:~/cloud/adam/xubo/data/test20160310$mkdir-p~/xubo/ref/GRCH38Index/hadoop@Master:~/cloud/adam/xubo/data/test20160310$mvGCA_000001405.15_GRCh38/*~/xubo/ref/GRCH38I
基因数据处理54之bwa-mem运行paird-end（1千万条100bp的reads） KeepLearningBigData 基因数据处理 Spark问题
指令：```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20Paired1.fqg38L100c10000000Nhs20Paired2.fq>g38L100c10000000Nhs20Paired12
基因数据处理1之mapping_to_cram KeepLearningBigData 基因数据处理
基因数据处理1之mapping_to_cram参考资料：AWorkedExampleObtainsomepublicdataWewillusethefirst100,000read-pairsfromayeastdataset.curlftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR507/SRR507778/SRR507778_1.fastq.gz|gzip-d|he
基因数据处理72之GATK安装成功 KeepLearningBigData 基因数据处理
1.下载：gitclonehttps://github.com/broadgsa/gatk-protected.git2.安装：gitcheckout3.5mvncleanpackage-DskipTests3.安装成功：[INFO]ReactorSummary:[INFO][INFO]GATKRoot..........................................SUCCES
基因数据处理77之从vcf文件中提取某条染色体的数据 KeepLearningBigData 基因数据处理
1.代码：/***@authorxubo*/packageorg.gcdss.cli.vcfimportorg.apache.spark.{SparkConf,SparkContext}/***Createdbyxuboon2016/5/23.*/objectextractGRCH38chr20vcf{defmain(args:Array[String]){valconf=newSparkConf
基因数据处理34之使用samtools和bcftools进行变异分析 KeepLearningBigData 基因数据处理
1.指令：（1）samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_snp_A2G_chr20_225058.sorted.bam>NA12878_snp_A2G_chr20_225058.variants或者：samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_
基因数据处理116之重新运行SparkBWA Yarn集群模式 KeepLearningBigData 基因数据处理
更多代码请见：https://github.com/xubo245基因数据处理系列1.解释很久没运行SparkBWA了，系统文件有点多，重新运行。2.代码：endhadoop@Master:~/disk2/xubo/project/alignment/sparkBWA$vig38L100c100000Nhs20Paired12SparkBWAYarnPartition0.shecho"start"
基因数据处理117之重新多次运行SparkBWA Yarn集群 KeepLearningBigData 基因数据处理
更多代码请见：https://github.com/xubo245基因数据处理系列1.解释重新运行，跟换了文件地址后来终端，需要运行其他的2.代码：hadoop@Master:~/disk2/xubo/project/alignment/sparkBWA$visparkBWA.shforjin10000100000100000010000000doforiin50100doecho$iecho$j
基因数据处理112之运行gcdss的avocado编译识别报错getRecordGroupSample空指针异常解决办法 KeepLearningBigData 基因数据处理
前面一片已经讲了遇到的问题，之前也遇到过：基因数据处理31之avocado运行avocado-cli中的avocado问题3-变异识别找不到RecordGroupSample（null）http://blog.csdn.net/xubo245/article/details/51525241解决办法：在读入的sam/bam/adam数据进行判定，如果RecordGroup等数据为空，则加上默认值在
基因数据处理110之批量测试SparkBWA和CS-bwamem算法 KeepLearningBigData 基因数据处理
1.总的脚本hadoop@Master:~/xubo/project/alignment$catrunSparkBWAAndCsbwamem.shecho"startsparkBWA"./sparkBWA/sparkBWA.shecho"startCsbwamemalgin"./cs-bwamem/csbwamemAlign.shecho"startCsbwamemmerge"./cs-bwame
基因数据处理106之bwa-mem运行paird-end（1千万条100bp的reads g38L100c10000000Nhs20Paired12） KeepLearningBigData 基因数据处理
脚本：hadoop@Master:~/xubo/project/alignment/sparkBWA$catg38L100c10000000Nhs20Paired12Bwamem.shecho"start"startTime4=`date+"%s.%N"`time4=`date+"%Y%m%d%H%M%S"`#spark-submit--classorg.apache.spark.examples
基因数据处理113之对avocado识别的SparkBWA变异数据进行疾病分析_ KeepLearningBigData 基因数据处理
（一）问题问题1：avocado数据读取：avocado存到磁盘是：RDD[Genotype]valprocessedGenotypes:RDD[Genotype]=postProcessVariants(calledVariants,stats).flatMap(variantContext=>variantContext.genotypes)//savevariantstooutputfile
基因数据处理109之SparkBWA运行成功的部分log KeepLearningBigData 基因数据处理
数据来自：http://219.219.220.149:18080/history/application_1466866953605_0001/executors/http://219.219.220.248:8042/node/containerlogs/container_1466866953605_0001_01_000003/hadoop/stderr/?start=0SLF4J:Cla
Spark问题14之Spark stage retry问题 KeepLearningBigData Spark问题
更多代码请见：https://github.com/xubo245基因数据处理系列之SparkBWA1.解释1.1简述当partitions超过节点数量的时候Lostexecutor的问题，已经提交到SparkBWA中，https://github.com/citiususc/SparkBWA/issues/35另外发现，tmp里面有临时文件没有删除，而且stageretry未解决2.记录完整报错
基因数据处理102之SparkBWA本地运行100万条paired-reads实例 KeepLearningBigData 基因数据处理
脚本：spark-submit--classSparkBWA\--masterlocal\--archivesbwa.zip\SparkBWA.jar\-algorithmmem-readspaired\-index/home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta\-partitions3\/xubo/alignment/sp
基因数据处理101之SparkBWA本地运行配置和实例 KeepLearningBigData 基因数据处理
1.修改Makefile.common：将LIBBWA_LIBS=-lrt改为LIBBWA_LIBS=-lrt-lz不然会报错误【5】2.make之后修改java.library.path步骤：vi/etc/profile加入exportLD_LIBRARY_PATH=/home/hadoop/xubo/tools/SparkBWA/build:$LD_LIBRARY_PATH使生效：source
深入浅出Java Annotation(元注解和自定义注解） Josh_Persistence Java Annotation 元注解自定义注解
一、基本概述　　 Annontation是Java5开始引入的新特征。中文名称一般叫注解。它提供了一种安全的类似注释的机制，用来将任何的信息或元数据（metadata）与程序元素（类、方法、成员变量等）进行关联。　　更通俗的意思是为程序的元素（类、方法、成员变量）加上更直观更明了的说明，这些说明信息是与程序的业务逻辑无关，并且是供指定的工具或
mysql优化特定类型的查询 annan211 java 工作 mysql
本节所介绍的查询优化的技巧都是和特定版本相关的，所以对于未来mysql的版本未必适用。 1 优化count查询对于count这个函数的网上的大部分资料都是错误的或者是理解的都是一知半解的。在做优化之前我们先来看看真正的count()函数的作用到底是什么。 count()是一个特殊的函数，有两种非常不同的作用，他可以统计某个列值的数量，也可以统计行数。在统
MAC下安装多版本JDK和切换几种方式棋子chessman jdk
环境： MAC AIR,OS X 10.10,64位历史：过去 Mac 上的 Java 都是由 Apple 自己提供，只支持到 Java 6，并且OS X 10.7 开始系统并不自带（而是可选安装）（原自带的是1.6）。后来 Apple 加入 OpenJDK 继续支持 Java 6，而 Java 7 将由 Oracle 负责提供。在终端中输入jav
javaScript （1） Array_06 JavaScript java 浏览器
JavaScript 1、运算符　　运算符就是完成操作的一系列符号，它有七类：　　赋值运算符（=,+=,-=,*=,/=,%=,<<=,>>=,|=,&=）、算术运算符(+,-,*,/,++,--,%)、比较运算符(>,<,<=,>=,==,===,!=,!==)、逻辑运算符(||,&&,!)、条件运算(?:)、位
国内顶级代码分享网站袁潇含 java jdk oracle .net PHP
现在国内很多开源网站感觉都是为了利益而做的当然利益是肯定的,否则谁也不会免费的去做网站 &
Elasticsearch、MongoDB和Hadoop比较随意而生 mongodb hadoop 搜索引擎
IT界在过去几年中出现了一个有趣的现象。很多新的技术出现并立即拥抱了“大数据”。稍微老一点的技术也会将大数据添进自己的特性，避免落大部队太远，我们看到了不同技术之间的边际的模糊化。假如你有诸如Elasticsearch或者Solr这样的搜索引擎，它们存储着JSON文档，MongoDB存着JSON文档，或者一堆JSON文档存放在一个Hadoop集群的HDFS中。你可以使用这三种配
mac os 系统科研软件总结张亚雄 mac os
1.1 Microsoft Office for Mac 2011 大客户版，自行搜索。 1.2 Latex （MacTex）: 系统环境：https://tug.org/mactex/ &nb
Maven实战（四）生命周期 AdyZhang maven
1. 三套生命周期 Maven拥有三套相互独立的生命周期，它们分别为clean，default和site。每个生命周期包含一些阶段，这些阶段是有顺序的，并且后面的阶段依赖于前面的阶段，用户和Maven最直接的交互方式就是调用这些生命周期阶段。以clean生命周期为例，它包含的阶段有pre-clean, clean 和 post
Linux下Jenkins迁移 aijuans Jenkins
1. 将Jenkins程序目录copy过去源程序在/export/data/tomcatRoot/ofctest-jenkins.jd.com下面 tar -cvzf jenkins.tar.gz ofctest-jenkins.jd.com &
request.getInputStream()只能获取一次的问题 ayaoxinchao request Inputstream
问题：在使用HTTP协议实现应用间接口通信时，服务端读取客户端请求过来的数据，会用到request.getInputStream()，第一次读取的时候可以读取到数据，但是接下来的读取操作都读取不到数据原因： 1. 一个InputStream对象在被读取完成后，将无法被再次读取，始终返回-1； 2. InputStream并没有实现reset方法（可以重
数据库SQL优化大总结之百万级数据库优化方案 BigBird2012 SQL优化
网上关于SQL优化的教程很多，但是比较杂乱。近日有空整理了一下，写出来跟大家分享一下，其中有错误和不足的地方，还请大家纠正补充。这篇文章我花费了大量的时间查找资料、修改、排版，希望大家阅读之后，感觉好的话推荐给更多的人，让更多的人看到、纠正以及补充。 1.对查询进行优化，要尽量避免全表扫描，首先应考虑在 where 及 order by 涉及的列上建立索引。 2.应尽量避免在 where
jsonObject的使用 bijian1013 java json
在项目中难免会用java处理json格式的数据，因此封装了一个JSONUtil工具类。 JSONUtil.java package com.bijian.json.study; import java.util.ArrayList; import java.util.Date; import java.util.HashMap;
[Zookeeper学习笔记之六]Zookeeper源代码分析之Zookeeper.WatchRegistration bit1129 zookeeper
Zookeeper类是Zookeeper提供给用户访问Zookeeper service的主要API，它包含了如下几个内部类首先分析它的内部类，从WatchRegistration开始，为指定的znode path注册一个Watcher， /** * Register a watcher for a particular p
【Scala十三】Scala核心七：部分应用函数 bit1129 scala
何为部分应用函数？ Partially applied function: A function that’s used in an expression and that misses some of its arguments.For instance, if function f has type Int => Int => Int, then f and f(1) are p
Tomcat Error listenerStart 终极大法 ronin47 tomcat
Tomcat报的错太含糊了，什么错都没报出来，只提示了Error listenerStart。为了调试，我们要获得更详细的日志。可以在WEB-INF/classes目录下新建一个文件叫logging.properties，内容如下 Java代码 handlers = org.apache.juli.FileHandler, java.util.logging.ConsoleHa
不用加减符号实现加减法 BrokenDreams 实现
今天有群友发了一个问题，要求不用加减符号(包括负号)来实现加减法。分析一下，先看最简单的情况，假设1+1，按二进制算的话结果是10，可以看到从右往左的第一位变为0，第二位由于进位变为1。
读《研磨设计模式》-代码笔记-状态模式-State bylijinnan java 设计模式
声明：本文只为方便我个人查阅和理解，详细的分析以及源代码请移步原作者的博客http://chjavach.iteye.com/ /* 当一个对象的内在状态改变时允许改变其行为，这个对象看起来像是改变了其类状态模式主要解决的是当控制一个对象状态的条件表达式过于复杂时的情况把状态的判断逻辑转移到表示不同状态的一系列类中，可以把复杂的判断逻辑简化如果在
CUDA程序block和thread超出硬件允许值时的异常 cherishLC CUDA
调用CUDA的核函数时指定block 和 thread大小，该大小可以是dim3类型的（三维数组），只用一维时可以是usigned int型的。以下程序验证了当block或thread大小超出硬件允许值时会产生异常！！！GPU根本不会执行运算！！！所以验证结果的正确性很重要！！！在VS中创建CUDA项目会有一个模板，里面有更详细的状态验证。以下程序在K5000GPU上跑的。
诡异的超长时间GC问题定位 chenchao051 jvm cms GC hbase swap
HBase的GC策略采用PawNew+CMS, 这是大众化的配置，ParNew经常会出现停顿时间特别长的情况，有时候甚至长到令人发指的地步，例如请看如下日志： 2012-10-17T05:54:54.293+0800: 739594.224: [GC 739606.508: [ParNew: 996800K->110720K(996800K), 178.8826900 secs] 3700
maven环境快速搭建 daizj 安装 mavne 环境配置
一下载maven 安装maven之前，要先安装jdk及配置JAVA_HOME环境变量。这个安装和配置java环境不用多说。 maven下载地址：http://maven.apache.org/download.html，目前最新的是这个apache-maven-3.2.5-bin.zip，然后解压在任意位置，最好地址中不要带中文字符，这个做java 的都知道，地址中出现中文会出现很多
PHP网站安全，避免PHP网站受到攻击的方法 dcj3sjt126com PHP
对于PHP网站安全主要存在这样几种攻击方式:1、命令注入(Command Injection)2、eval注入(Eval Injection)3、客户端脚本攻击(Script Insertion)4、跨网站脚本攻击(Cross Site Scripting, XSS)5、SQL注入攻击(SQL injection)6、跨网站请求伪造攻击(Cross Site Request Forgerie
yii中给CGridView设置默认的排序根据时间倒序的方法 dcj3sjt126com GridView
public function searchWithRelated() { $criteria = new CDbCriteria; $criteria->together = true; //without th
Java集合对象和数组对象的转换 dyy_gusi java集合
在开发中，我们经常需要将集合对象（List，Set）转换为数组对象，或者将数组对象转换为集合对象。Java提供了相互转换的工具，但是我们使用的时候需要注意，不能乱用滥用。 1、数组对象转换为集合对象最暴力的方式是new一个集合对象，然后遍历数组，依次将数组中的元素放入到新的集合中，但是这样做显然过
nginx同一主机部署多个应用 geeksun nginx
近日有一需求，需要在一台主机上用nginx部署2个php应用，分别是wordpress和wiki，探索了半天，终于部署好了，下面把过程记录下来。 1. 在nginx下创建vhosts目录，用以放置vhost文件。 mkdir vhosts 2. 修改nginx.conf的配置，在http节点增加下面内容设置，用来包含vhosts里的配置文件 #
ubuntu添加admin权限的用户账号 hongtoushizi ubuntu useradd
ubuntu创建账号的方式通常用到两种：useradd 和adduser . 本人尝试了useradd方法，步骤如下： 1:useradd 使用useradd时，如果后面不加任何参数的话，如：sudo useradd sysadm 创建出来的用户将是默认的三无用户：无home directory ,无密码,无系统shell。顾应该如下操作：
第五章常用Lua开发库2-JSON库、编码转换、字符串处理 jinnianshilongnian nginx lua
JSON库在进行数据传输时JSON格式目前应用广泛，因此从Lua对象与JSON字符串之间相互转换是一个非常常见的功能；目前Lua也有几个JSON库，本人用过cjson、dkjson。其中cjson的语法严格（比如unicode \u0020\u7eaf），要求符合规范否则会解析失败（如\u002），而dkjson相对宽松，当然也可以通过修改cjson的源码来完成
Spring定时器配置的两种实现方式OpenSymphony Quartz和java Timer详解 yaerfeng1989 timer quartz 定时器
原创整理不易，转载请注明出处：Spring定时器配置的两种实现方式OpenSymphony Quartz和java Timer详解代码下载地址：http://www.zuidaima.com/share/1772648445103104.htm 有两种流行Spring定时器配置：Java的Timer类和OpenSymphony的Quartz。 1.Java Timer定时首先继承jav
Linux下df与du两个命令的差别？ pda158 linux
　一、df显示文件系统的使用情况，与du比較，就是更全盘化。　　最经常使用的就是 df -T，显示文件系统的使用情况并显示文件系统的类型。　　举比例如以下：　　[root@localhost ~]# df -T 　　Filesystem Type &n
[转]SQLite的工具类 ---- 通过反射把Cursor封装到VO对象 ctfzh VO android sqlite 反射 Cursor
在写DAO层时，觉得从Cursor里一个一个的取出字段值再装到VO(值对象)里太麻烦了，就写了一个工具类，用到了反射，可以把查询记录的值装到对应的VO里，也可以生成该VO的List。使用时需要注意：考虑到Android的性能问题，VO没有使用Setter和Getter，而是直接用public的属性。表中的字段名需要和VO的属性名一样，要是不一样就得在查询的SQL中
该学习笔记用到的Employee表 vipbooks oracle sql 工作
这是我在学习Oracle是用到的Employee表，在该笔记中用到的就是这张表，大家可以用它来学习和练习。 drop table Employee; -- 员工信息表 create table Employee( -- 员工编号 EmpNo number(3) primary key, -- 姓