xiaocaichonga

[置顶] Hadoop 实战之分析专利引用数据集（二）

环境：Vmware 8.0 和Ubuntu11.04

Hadoop 实战之分析专利引用数据集（二）---计算专利引用数据并排序

第一步：首先创建一个工程命名为HadoopTest.目录结构如下图:

第二步: 在/home/tanglg1987目录下新建一个start.sh脚本文件，每次启动虚拟机都要删除/tmp目录下的全部文件，重新格式化namenode，代码如下：

sudo rm -rf /tmp/*
rm -rf /home/tanglg1987/hadoop-0.20.2/logs
hadoop namenode -format
hadoop datanode -format
start-all.sh
hadoop fs -mkdir input 
hadoop dfsadmin -safemode leave

第三步：给start.sh增加执行权限并启动hadoop伪分布式集群，代码如下:

chmod 777 /home/tanglg1987/ start.sh
./start.sh

执行过程如下：

第四步：上传本地文件到hdfs

在专利局http://data.nber.org/patents/网站下载专利数据

http://data.nber.org/patents/apat63_99.zip

hadoop fs -put /home/tanglg1987/cite75_99.txt input

五步：新建一个MyJob2.java，代码如下：

package com.baison.action;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class MyJob2 extends Configured implements Tool {  
    public static class MapClass extends Mapper<LongWritable, Text, Text, Text> {       
        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {                      
            String[] citation = value.toString().split(",");
            context.write(new Text(citation[1]), new Text(citation[0]));
        }
    }    
    public static class Reduce extends Reducer<Text, Text, Text, Text> {       
        public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {                         
            String csv = "";
            for (Text val:values) {
                if (csv.length() > 0) csv += ",";
                csv += val.toString();
            }         
            context.write(key, new Text(csv));
        }
    }
    
    public int run(String[] args) throws Exception {
        Configuration conf = getConf();   
        Job job = new Job(conf, "MyJob");
        job.setJarByClass(MyJob.class);     
        Path in = new Path(args[0]);
        Path out = new Path(args[1]);
        FileInputFormat.setInputPaths(job, in);
        FileOutputFormat.setOutputPath(job, out);   
        job.setMapperClass(MapClass.class);
        job.setReducerClass(Reduce.class);    
        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);       
        System.exit(job.waitForCompletion(true)?0:1);    
        return 0;
    }  
    public static void main(String[] args) throws Exception { 
    	String [] arg={"hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt","hdfs://localhost:9100/user/tanglg1987/output"};
        int res = ToolRunner.run(new Configuration(), new MyJob(), arg);        
        System.exit(res);
    }
}

第六步：Run On Hadoop，运行过程如下：

hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt
hdfs://localhost:9100/user/tanglg1987/output
12/10/18 22:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 22:17:45 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 22:17:46 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:17:46 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 22:17:46 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:17:46 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:17:46 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:17:46 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:17:46 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:17:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:47 INFO mapred.MapTask: bufstart = 0; bufend = 4185926; bufvoid = 99614720
12/10/18 22:17:47 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:17:47 INFO mapred.JobClient: map 0% reduce 0%
12/10/18 22:17:48 INFO mapred.MapTask: Finished spill 0
12/10/18 22:17:48 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:48 INFO mapred.MapTask: bufstart = 4185926; bufend = 8372612; bufvoid = 99614720
12/10/18 22:17:48 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:17:49 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:49 INFO mapred.MapTask: Finished spill 1
12/10/18 22:17:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:50 INFO mapred.MapTask: bufstart = 8372612; bufend = 12558934; bufvoid = 99614720
12/10/18 22:17:50 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:17:50 INFO mapred.JobClient: map 14% reduce 0%
12/10/18 22:17:51 INFO mapred.MapTask: Finished spill 2
12/10/18 22:17:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:51 INFO mapred.MapTask: bufstart = 12558934; bufend = 16745318; bufvoid = 99614720
12/10/18 22:17:51 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:17:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:52 INFO mapred.MapTask: Finished spill 3
12/10/18 22:17:52 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:52 INFO mapred.MapTask: bufstart = 16745318; bufend = 20931889; bufvoid = 99614720
12/10/18 22:17:52 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:17:53 INFO mapred.JobClient: map 26% reduce 0%
12/10/18 22:17:53 INFO mapred.MapTask: Finished spill 4
12/10/18 22:17:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:54 INFO mapred.MapTask: bufstart = 20931889; bufend = 25118734; bufvoid = 99614720
12/10/18 22:17:54 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:17:55 INFO mapred.MapTask: Finished spill 5
12/10/18 22:17:55 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:55 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:55 INFO mapred.MapTask: bufstart = 25118734; bufend = 29305269; bufvoid = 99614720
12/10/18 22:17:55 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:17:56 INFO mapred.JobClient: map 42% reduce 0%
12/10/18 22:17:56 INFO mapred.MapTask: Finished spill 6
12/10/18 22:17:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:56 INFO mapred.MapTask: bufstart = 29305269; bufend = 33492764; bufvoid = 99614720
12/10/18 22:17:56 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:17:57 INFO mapred.MapTask: Finished spill 7
12/10/18 22:17:57 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:57 INFO mapred.MapTask: bufstart = 33492764; bufend = 37680391; bufvoid = 99614720
12/10/18 22:17:57 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:17:58 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:58 INFO mapred.MapTask: Finished spill 8
12/10/18 22:17:59 INFO mapred.JobClient: map 57% reduce 0%
12/10/18 22:17:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:59 INFO mapred.MapTask: bufstart = 37680391; bufend = 41868206; bufvoid = 99614720
12/10/18 22:17:59 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:18:00 INFO mapred.MapTask: Finished spill 9
12/10/18 22:18:00 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:00 INFO mapred.MapTask: bufstart = 41868206; bufend = 46056257; bufvoid = 99614720
12/10/18 22:18:00 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:18:01 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:02 INFO mapred.MapTask: Finished spill 10
12/10/18 22:18:02 INFO mapred.JobClient: map 70% reduce 0%
12/10/18 22:18:02 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:02 INFO mapred.MapTask: bufstart = 46056257; bufend = 50244288; bufvoid = 99614720
12/10/18 22:18:02 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:18:04 INFO mapred.MapTask: Finished spill 11
12/10/18 22:18:04 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:04 INFO mapred.MapTask: bufstart = 50244288; bufend = 54432271; bufvoid = 99614720
12/10/18 22:18:04 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:18:04 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:05 INFO mapred.JobClient: map 82% reduce 0%
12/10/18 22:18:05 INFO mapred.MapTask: Finished spill 12
12/10/18 22:18:07 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:07 INFO mapred.MapTask: bufstart = 54432271; bufend = 58620635; bufvoid = 99614720
12/10/18 22:18:07 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:18:07 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:07 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 22:18:08 INFO mapred.MapTask: Finished spill 13
12/10/18 22:18:08 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:08 INFO mapred.MapTask: bufstart = 58620635; bufend = 62808941; bufvoid = 99614720
12/10/18 22:18:08 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:18:10 INFO mapred.MapTask: Finished spill 14
12/10/18 22:18:10 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:10 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:10 INFO mapred.MapTask: bufstart = 62808941; bufend = 66997060; bufvoid = 99614720
12/10/18 22:18:10 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:18:10 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:18:10 INFO mapred.JobClient: map 99% reduce 0%
12/10/18 22:18:11 INFO mapred.MapTask: Finished spill 15
12/10/18 22:18:11 INFO mapred.MapTask: Finished spill 16
12/10/18 22:18:11 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:18:11 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:18:13 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:13 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:18:15 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75511476 bytes
12/10/18 22:18:16 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:19 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:21 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 22:18:21 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:21 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 22:18:22 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:18:22 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:18:22 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:18:22 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:18:22 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:22 INFO mapred.MapTask: bufstart = 0; bufend = 4188247; bufvoid = 99614720
12/10/18 22:18:22 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:18:23 INFO mapred.MapTask: Finished spill 0
12/10/18 22:18:23 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:23 INFO mapred.MapTask: bufstart = 4188247; bufend = 8376832; bufvoid = 99614720
12/10/18 22:18:23 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:18:24 INFO mapred.MapTask: Finished spill 1
12/10/18 22:18:25 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:25 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:25 INFO mapred.MapTask: bufstart = 8376832; bufend = 12565282; bufvoid = 99614720
12/10/18 22:18:25 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:18:25 INFO mapred.JobClient: map 58% reduce 0%
12/10/18 22:18:26 INFO mapred.MapTask: Finished spill 2
12/10/18 22:18:26 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:26 INFO mapred.MapTask: bufstart = 12565282; bufend = 16754414; bufvoid = 99614720
12/10/18 22:18:26 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:18:27 INFO mapred.MapTask: Finished spill 3
12/10/18 22:18:27 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:27 INFO mapred.MapTask: bufstart = 16754414; bufend = 20943273; bufvoid = 99614720
12/10/18 22:18:27 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:18:28 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:28 INFO mapred.JobClient: map 66% reduce 0%
12/10/18 22:18:29 INFO mapred.MapTask: Finished spill 4
12/10/18 22:18:30 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:30 INFO mapred.MapTask: bufstart = 20943273; bufend = 25132396; bufvoid = 99614720
12/10/18 22:18:30 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:18:31 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:31 INFO mapred.MapTask: Finished spill 5
12/10/18 22:18:31 INFO mapred.JobClient: map 69% reduce 0%
12/10/18 22:18:32 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:32 INFO mapred.MapTask: bufstart = 25132396; bufend = 29321431; bufvoid = 99614720
12/10/18 22:18:32 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:18:34 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:34 INFO mapred.MapTask: Finished spill 6
12/10/18 22:18:34 INFO mapred.JobClient: map 72% reduce 0%
12/10/18 22:18:35 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:35 INFO mapred.MapTask: bufstart = 29321431; bufend = 33510738; bufvoid = 99614720
12/10/18 22:18:35 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:18:36 INFO mapred.MapTask: Finished spill 7
12/10/18 22:18:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:37 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:37 INFO mapred.MapTask: bufstart = 33510738; bufend = 37700146; bufvoid = 99614720
12/10/18 22:18:37 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:18:37 INFO mapred.JobClient: map 75% reduce 0%
12/10/18 22:18:38 INFO mapred.MapTask: Finished spill 8
12/10/18 22:18:39 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:39 INFO mapred.MapTask: bufstart = 37700146; bufend = 41889795; bufvoid = 99614720
12/10/18 22:18:39 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:18:40 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:40 INFO mapred.JobClient: map 81% reduce 0%
12/10/18 22:18:41 INFO mapred.MapTask: Finished spill 9
12/10/18 22:18:41 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:41 INFO mapred.MapTask: bufstart = 41889795; bufend = 46079341; bufvoid = 99614720
12/10/18 22:18:41 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:18:43 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:43 INFO mapred.MapTask: Finished spill 10
12/10/18 22:18:43 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:43 INFO mapred.MapTask: bufstart = 46079341; bufend = 50269061; bufvoid = 99614720
12/10/18 22:18:43 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:18:43 INFO mapred.JobClient: map 85% reduce 0%
12/10/18 22:18:44 INFO mapred.MapTask: Finished spill 11
12/10/18 22:18:45 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:45 INFO mapred.MapTask: bufstart = 50269061; bufend = 54458860; bufvoid = 99614720
12/10/18 22:18:45 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:18:46 INFO mapred.MapTask: Finished spill 12
12/10/18 22:18:46 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:46 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:46 INFO mapred.MapTask: bufstart = 54458860; bufend = 58648475; bufvoid = 99614720
12/10/18 22:18:46 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:18:46 INFO mapred.JobClient: map 91% reduce 0%
12/10/18 22:18:47 INFO mapred.MapTask: Finished spill 13
12/10/18 22:18:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:47 INFO mapred.MapTask: bufstart = 58648475; bufend = 62838648; bufvoid = 99614720
12/10/18 22:18:47 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:18:48 INFO mapred.MapTask: Finished spill 14
12/10/18 22:18:48 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:48 INFO mapred.MapTask: bufstart = 62838648; bufend = 67028992; bufvoid = 99614720
12/10/18 22:18:48 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:18:48 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:18:49 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:49 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:18:49 INFO mapred.MapTask: Finished spill 15
12/10/18 22:18:49 INFO mapred.MapTask: Finished spill 16
12/10/18 22:18:49 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:18:49 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:18:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:52 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75507463 bytes
12/10/18 22:18:55 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:58 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:59 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
12/10/18 22:18:59 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:59 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
12/10/18 22:19:00 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:19:00 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:19:00 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:19:00 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:19:00 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:00 INFO mapred.MapTask: bufstart = 0; bufend = 4190341; bufvoid = 99614720
12/10/18 22:19:00 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:19:01 INFO mapred.MapTask: Finished spill 0
12/10/18 22:19:01 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:01 INFO mapred.MapTask: bufstart = 4190341; bufend = 8380745; bufvoid = 99614720
12/10/18 22:19:01 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:19:02 INFO mapred.MapTask: Finished spill 1
12/10/18 22:19:03 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:03 INFO mapred.MapTask: bufstart = 8380745; bufend = 12571307; bufvoid = 99614720
12/10/18 22:19:03 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:19:03 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:03 INFO mapred.JobClient: map 73% reduce 0%
12/10/18 22:19:04 INFO mapred.MapTask: Finished spill 2
12/10/18 22:19:04 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:04 INFO mapred.MapTask: bufstart = 12571307; bufend = 16762033; bufvoid = 99614720
12/10/18 22:19:04 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:19:05 INFO mapred.MapTask: Finished spill 3
12/10/18 22:19:06 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:06 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:06 INFO mapred.MapTask: bufstart = 16762033; bufend = 20952800; bufvoid = 99614720
12/10/18 22:19:06 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:19:06 INFO mapred.JobClient: map 76% reduce 0%
12/10/18 22:19:06 INFO mapred.MapTask: Finished spill 4
12/10/18 22:19:07 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:07 INFO mapred.MapTask: bufstart = 20952800; bufend = 25143412; bufvoid = 99614720
12/10/18 22:19:07 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:19:08 INFO mapred.MapTask: Finished spill 5
12/10/18 22:19:08 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:08 INFO mapred.MapTask: bufstart = 25143412; bufend = 29334124; bufvoid = 99614720
12/10/18 22:19:08 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:19:09 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:09 INFO mapred.MapTask: Finished spill 6
12/10/18 22:19:09 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:09 INFO mapred.MapTask: bufstart = 29334124; bufend = 33524778; bufvoid = 99614720
12/10/18 22:19:09 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:19:09 INFO mapred.JobClient: map 81% reduce 0%
12/10/18 22:19:10 INFO mapred.MapTask: Finished spill 7
12/10/18 22:19:10 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:10 INFO mapred.MapTask: bufstart = 33524778; bufend = 37715716; bufvoid = 99614720
12/10/18 22:19:10 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:19:11 INFO mapred.MapTask: Finished spill 8
12/10/18 22:19:11 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:11 INFO mapred.MapTask: bufstart = 37715716; bufend = 41906544; bufvoid = 99614720
12/10/18 22:19:11 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:19:12 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:12 INFO mapred.MapTask: Finished spill 9
12/10/18 22:19:12 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 22:19:12 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:12 INFO mapred.MapTask: bufstart = 41906544; bufend = 46097751; bufvoid = 99614720
12/10/18 22:19:12 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:19:13 INFO mapred.MapTask: Finished spill 10
12/10/18 22:19:13 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:13 INFO mapred.MapTask: bufstart = 46097751; bufend = 50288897; bufvoid = 99614720
12/10/18 22:19:13 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:19:14 INFO mapred.MapTask: Finished spill 11
12/10/18 22:19:14 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:14 INFO mapred.MapTask: bufstart = 50288897; bufend = 54480358; bufvoid = 99614720
12/10/18 22:19:14 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:19:15 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:15 INFO mapred.MapTask: Finished spill 12
12/10/18 22:19:15 INFO mapred.JobClient: map 94% reduce 0%
12/10/18 22:19:15 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:15 INFO mapred.MapTask: bufstart = 54480358; bufend = 58671642; bufvoid = 99614720
12/10/18 22:19:15 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:19:16 INFO mapred.MapTask: Finished spill 13
12/10/18 22:19:16 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:16 INFO mapred.MapTask: bufstart = 58671642; bufend = 62863227; bufvoid = 99614720
12/10/18 22:19:16 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:19:17 INFO mapred.MapTask: Finished spill 14
12/10/18 22:19:17 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:17 INFO mapred.MapTask: bufstart = 62863227; bufend = 67054803; bufvoid = 99614720
12/10/18 22:19:17 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:19:17 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:19:18 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:18 INFO mapred.MapTask: Finished spill 15
12/10/18 22:19:18 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:19:19 INFO mapred.MapTask: Finished spill 16
12/10/18 22:19:19 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:19:19 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:19:21 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:21 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75504227 bytes
12/10/18 22:19:24 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:27 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:27 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
12/10/18 22:19:27 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:27 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000002_0' done.
12/10/18 22:19:27 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:19:27 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:19:28 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:19:28 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:19:28 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:28 INFO mapred.MapTask: bufstart = 0; bufend = 4191578; bufvoid = 99614720
12/10/18 22:19:28 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:19:29 INFO mapred.MapTask: Finished spill 0
12/10/18 22:19:29 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:29 INFO mapred.MapTask: bufstart = 4191578; bufend = 8383078; bufvoid = 99614720
12/10/18 22:19:29 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:19:30 INFO mapred.MapTask: Finished spill 1
12/10/18 22:19:30 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:30 INFO mapred.MapTask: bufstart = 8383078; bufend = 12574490; bufvoid = 99614720
12/10/18 22:19:30 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:19:31 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:31 INFO mapred.MapTask: Finished spill 2
12/10/18 22:19:31 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:31 INFO mapred.MapTask: bufstart = 12574490; bufend = 16766127; bufvoid = 99614720
12/10/18 22:19:31 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:19:31 INFO mapred.JobClient: map 80% reduce 0%
12/10/18 22:19:32 INFO mapred.MapTask: Finished spill 3
12/10/18 22:19:32 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:32 INFO mapred.MapTask: bufstart = 16766127; bufend = 20958217; bufvoid = 99614720
12/10/18 22:19:32 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:19:33 INFO mapred.MapTask: Finished spill 4
12/10/18 22:19:33 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:33 INFO mapred.MapTask: bufstart = 20958217; bufend = 25150065; bufvoid = 99614720
12/10/18 22:19:33 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:19:34 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:34 INFO mapred.MapTask: Finished spill 5
12/10/18 22:19:34 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:34 INFO mapred.MapTask: bufstart = 25150065; bufend = 29341612; bufvoid = 99614720
12/10/18 22:19:34 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:19:34 INFO mapred.JobClient: map 85% reduce 0%
12/10/18 22:19:35 INFO mapred.MapTask: Finished spill 6
12/10/18 22:19:35 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:35 INFO mapred.MapTask: bufstart = 29341612; bufend = 33533322; bufvoid = 99614720
12/10/18 22:19:35 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:19:36 INFO mapred.MapTask: Finished spill 7
12/10/18 22:19:36 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:36 INFO mapred.MapTask: bufstart = 33533322; bufend = 37725421; bufvoid = 99614720
12/10/18 22:19:36 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:19:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:37 INFO mapred.MapTask: Finished spill 8
12/10/18 22:19:37 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:37 INFO mapred.MapTask: bufstart = 37725421; bufend = 41917251; bufvoid = 99614720
12/10/18 22:19:37 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:19:37 INFO mapred.JobClient: map 90% reduce 0%
12/10/18 22:19:38 INFO mapred.MapTask: Finished spill 9
12/10/18 22:19:38 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:38 INFO mapred.MapTask: bufstart = 41917251; bufend = 46108919; bufvoid = 99614720
12/10/18 22:19:38 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:19:39 INFO mapred.MapTask: Finished spill 10
12/10/18 22:19:39 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:39 INFO mapred.MapTask: bufstart = 46108919; bufend = 50300567; bufvoid = 99614720
12/10/18 22:19:39 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:19:40 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:40 INFO mapred.MapTask: Finished spill 11
12/10/18 22:19:40 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:40 INFO mapred.MapTask: bufstart = 50300567; bufend = 54492159; bufvoid = 99614720
12/10/18 22:19:40 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:19:40 INFO mapred.JobClient: map 95% reduce 0%
12/10/18 22:19:41 INFO mapred.MapTask: Finished spill 12
12/10/18 22:19:41 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:41 INFO mapred.MapTask: bufstart = 54492159; bufend = 58684086; bufvoid = 99614720
12/10/18 22:19:41 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:19:42 INFO mapred.MapTask: Finished spill 13
12/10/18 22:19:42 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:19:43 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:43 INFO mapred.MapTask: Finished spill 14
12/10/18 22:19:43 INFO mapred.Merger: Merging 15 sorted segments
12/10/18 22:19:43 INFO mapred.Merger: Merging 6 intermediate segments out of a total of 15
12/10/18 22:19:43 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:19:46 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 70597223 bytes
12/10/18 22:19:46 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:49 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:52 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
12/10/18 22:19:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:52 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000003_0' done.
12/10/18 22:19:52 INFO mapred.LocalJobRunner:
12/10/18 22:19:52 INFO mapred.Merger: Merging 4 sorted segments
12/10/18 22:19:53 INFO mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 297120317 bytes
12/10/18 22:19:53 INFO mapred.LocalJobRunner:
12/10/18 22:19:58 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:19:58 INFO mapred.JobClient: map 100% reduce 69%
12/10/18 22:20:01 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:01 INFO mapred.JobClient: map 100% reduce 72%
12/10/18 22:20:04 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:04 INFO mapred.JobClient: map 100% reduce 74%
12/10/18 22:20:07 INFO mapredhdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt
hdfs://localhost:9100/user/tanglg1987/output
12/10/18 22:17:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/10/18 22:17:45 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
12/10/18 22:17:46 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:17:46 INFO mapred.JobClient: Running job: job_local_0001
12/10/18 22:17:46 INFO mapred.FileInputFormat: Total input paths to process : 1
12/10/18 22:17:46 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:17:46 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:17:46 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:17:46 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:17:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:47 INFO mapred.MapTask: bufstart = 0; bufend = 4185926; bufvoid = 99614720
12/10/18 22:17:47 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:17:47 INFO mapred.JobClient: map 0% reduce 0%
12/10/18 22:17:48 INFO mapred.MapTask: Finished spill 0
12/10/18 22:17:48 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:48 INFO mapred.MapTask: bufstart = 4185926; bufend = 8372612; bufvoid = 99614720
12/10/18 22:17:48 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:17:49 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:49 INFO mapred.MapTask: Finished spill 1
12/10/18 22:17:50 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:50 INFO mapred.MapTask: bufstart = 8372612; bufend = 12558934; bufvoid = 99614720
12/10/18 22:17:50 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:17:50 INFO mapred.JobClient: map 14% reduce 0%
12/10/18 22:17:51 INFO mapred.MapTask: Finished spill 2
12/10/18 22:17:51 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:51 INFO mapred.MapTask: bufstart = 12558934; bufend = 16745318; bufvoid = 99614720
12/10/18 22:17:51 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:17:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:52 INFO mapred.MapTask: Finished spill 3
12/10/18 22:17:52 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:52 INFO mapred.MapTask: bufstart = 16745318; bufend = 20931889; bufvoid = 99614720
12/10/18 22:17:52 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:17:53 INFO mapred.JobClient: map 26% reduce 0%
12/10/18 22:17:53 INFO mapred.MapTask: Finished spill 4
12/10/18 22:17:54 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:54 INFO mapred.MapTask: bufstart = 20931889; bufend = 25118734; bufvoid = 99614720
12/10/18 22:17:54 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:17:55 INFO mapred.MapTask: Finished spill 5
12/10/18 22:17:55 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:55 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:55 INFO mapred.MapTask: bufstart = 25118734; bufend = 29305269; bufvoid = 99614720
12/10/18 22:17:55 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:17:56 INFO mapred.JobClient: map 42% reduce 0%
12/10/18 22:17:56 INFO mapred.MapTask: Finished spill 6
12/10/18 22:17:56 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:56 INFO mapred.MapTask: bufstart = 29305269; bufend = 33492764; bufvoid = 99614720
12/10/18 22:17:56 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:17:57 INFO mapred.MapTask: Finished spill 7
12/10/18 22:17:57 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:57 INFO mapred.MapTask: bufstart = 33492764; bufend = 37680391; bufvoid = 99614720
12/10/18 22:17:57 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:17:58 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:17:58 INFO mapred.MapTask: Finished spill 8
12/10/18 22:17:59 INFO mapred.JobClient: map 57% reduce 0%
12/10/18 22:17:59 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:17:59 INFO mapred.MapTask: bufstart = 37680391; bufend = 41868206; bufvoid = 99614720
12/10/18 22:17:59 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:18:00 INFO mapred.MapTask: Finished spill 9
12/10/18 22:18:00 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:00 INFO mapred.MapTask: bufstart = 41868206; bufend = 46056257; bufvoid = 99614720
12/10/18 22:18:00 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:18:01 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:02 INFO mapred.MapTask: Finished spill 10
12/10/18 22:18:02 INFO mapred.JobClient: map 70% reduce 0%
12/10/18 22:18:02 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:02 INFO mapred.MapTask: bufstart = 46056257; bufend = 50244288; bufvoid = 99614720
12/10/18 22:18:02 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:18:04 INFO mapred.MapTask: Finished spill 11
12/10/18 22:18:04 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:04 INFO mapred.MapTask: bufstart = 50244288; bufend = 54432271; bufvoid = 99614720
12/10/18 22:18:04 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:18:04 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:05 INFO mapred.JobClient: map 82% reduce 0%
12/10/18 22:18:05 INFO mapred.MapTask: Finished spill 12
12/10/18 22:18:07 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:07 INFO mapred.MapTask: bufstart = 54432271; bufend = 58620635; bufvoid = 99614720
12/10/18 22:18:07 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:18:07 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:07 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 22:18:08 INFO mapred.MapTask: Finished spill 13
12/10/18 22:18:08 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:08 INFO mapred.MapTask: bufstart = 58620635; bufend = 62808941; bufvoid = 99614720
12/10/18 22:18:08 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:18:10 INFO mapred.MapTask: Finished spill 14
12/10/18 22:18:10 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:10 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:10 INFO mapred.MapTask: bufstart = 62808941; bufend = 66997060; bufvoid = 99614720
12/10/18 22:18:10 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:18:10 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:18:10 INFO mapred.JobClient: map 99% reduce 0%
12/10/18 22:18:11 INFO mapred.MapTask: Finished spill 15
12/10/18 22:18:11 INFO mapred.MapTask: Finished spill 16
12/10/18 22:18:11 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:18:11 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:18:13 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:13 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:18:15 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75511476 bytes
12/10/18 22:18:16 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:19 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:21 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/10/18 22:18:21 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:0+67108864
12/10/18 22:18:21 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
12/10/18 22:18:22 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:18:22 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:18:22 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:18:22 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:18:22 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:22 INFO mapred.MapTask: bufstart = 0; bufend = 4188247; bufvoid = 99614720
12/10/18 22:18:22 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:18:23 INFO mapred.MapTask: Finished spill 0
12/10/18 22:18:23 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:23 INFO mapred.MapTask: bufstart = 4188247; bufend = 8376832; bufvoid = 99614720
12/10/18 22:18:23 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:18:24 INFO mapred.MapTask: Finished spill 1
12/10/18 22:18:25 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:25 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:25 INFO mapred.MapTask: bufstart = 8376832; bufend = 12565282; bufvoid = 99614720
12/10/18 22:18:25 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:18:25 INFO mapred.JobClient: map 58% reduce 0%
12/10/18 22:18:26 INFO mapred.MapTask: Finished spill 2
12/10/18 22:18:26 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:26 INFO mapred.MapTask: bufstart = 12565282; bufend = 16754414; bufvoid = 99614720
12/10/18 22:18:26 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:18:27 INFO mapred.MapTask: Finished spill 3
12/10/18 22:18:27 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:27 INFO mapred.MapTask: bufstart = 16754414; bufend = 20943273; bufvoid = 99614720
12/10/18 22:18:27 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:18:28 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:28 INFO mapred.JobClient: map 66% reduce 0%
12/10/18 22:18:29 INFO mapred.MapTask: Finished spill 4
12/10/18 22:18:30 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:30 INFO mapred.MapTask: bufstart = 20943273; bufend = 25132396; bufvoid = 99614720
12/10/18 22:18:30 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:18:31 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:31 INFO mapred.MapTask: Finished spill 5
12/10/18 22:18:31 INFO mapred.JobClient: map 69% reduce 0%
12/10/18 22:18:32 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:32 INFO mapred.MapTask: bufstart = 25132396; bufend = 29321431; bufvoid = 99614720
12/10/18 22:18:32 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:18:34 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:34 INFO mapred.MapTask: Finished spill 6
12/10/18 22:18:34 INFO mapred.JobClient: map 72% reduce 0%
12/10/18 22:18:35 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:35 INFO mapred.MapTask: bufstart = 29321431; bufend = 33510738; bufvoid = 99614720
12/10/18 22:18:35 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:18:36 INFO mapred.MapTask: Finished spill 7
12/10/18 22:18:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:37 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:37 INFO mapred.MapTask: bufstart = 33510738; bufend = 37700146; bufvoid = 99614720
12/10/18 22:18:37 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:18:37 INFO mapred.JobClient: map 75% reduce 0%
12/10/18 22:18:38 INFO mapred.MapTask: Finished spill 8
12/10/18 22:18:39 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:39 INFO mapred.MapTask: bufstart = 37700146; bufend = 41889795; bufvoid = 99614720
12/10/18 22:18:39 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:18:40 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:40 INFO mapred.JobClient: map 81% reduce 0%
12/10/18 22:18:41 INFO mapred.MapTask: Finished spill 9
12/10/18 22:18:41 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:41 INFO mapred.MapTask: bufstart = 41889795; bufend = 46079341; bufvoid = 99614720
12/10/18 22:18:41 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:18:43 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:43 INFO mapred.MapTask: Finished spill 10
12/10/18 22:18:43 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:43 INFO mapred.MapTask: bufstart = 46079341; bufend = 50269061; bufvoid = 99614720
12/10/18 22:18:43 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:18:43 INFO mapred.JobClient: map 85% reduce 0%
12/10/18 22:18:44 INFO mapred.MapTask: Finished spill 11
12/10/18 22:18:45 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:45 INFO mapred.MapTask: bufstart = 50269061; bufend = 54458860; bufvoid = 99614720
12/10/18 22:18:45 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:18:46 INFO mapred.MapTask: Finished spill 12
12/10/18 22:18:46 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:46 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:46 INFO mapred.MapTask: bufstart = 54458860; bufend = 58648475; bufvoid = 99614720
12/10/18 22:18:46 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:18:46 INFO mapred.JobClient: map 91% reduce 0%
12/10/18 22:18:47 INFO mapred.MapTask: Finished spill 13
12/10/18 22:18:47 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:47 INFO mapred.MapTask: bufstart = 58648475; bufend = 62838648; bufvoid = 99614720
12/10/18 22:18:47 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:18:48 INFO mapred.MapTask: Finished spill 14
12/10/18 22:18:48 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:18:48 INFO mapred.MapTask: bufstart = 62838648; bufend = 67028992; bufvoid = 99614720
12/10/18 22:18:48 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:18:48 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:18:49 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:49 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:18:49 INFO mapred.MapTask: Finished spill 15
12/10/18 22:18:49 INFO mapred.MapTask: Finished spill 16
12/10/18 22:18:49 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:18:49 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:18:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:52 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75507463 bytes
12/10/18 22:18:55 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:58 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:59 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
12/10/18 22:18:59 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:67108864+67108864
12/10/18 22:18:59 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
12/10/18 22:19:00 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:19:00 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:19:00 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:19:00 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:19:00 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:00 INFO mapred.MapTask: bufstart = 0; bufend = 4190341; bufvoid = 99614720
12/10/18 22:19:00 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:19:01 INFO mapred.MapTask: Finished spill 0
12/10/18 22:19:01 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:01 INFO mapred.MapTask: bufstart = 4190341; bufend = 8380745; bufvoid = 99614720
12/10/18 22:19:01 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:19:02 INFO mapred.MapTask: Finished spill 1
12/10/18 22:19:03 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:03 INFO mapred.MapTask: bufstart = 8380745; bufend = 12571307; bufvoid = 99614720
12/10/18 22:19:03 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:19:03 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:03 INFO mapred.JobClient: map 73% reduce 0%
12/10/18 22:19:04 INFO mapred.MapTask: Finished spill 2
12/10/18 22:19:04 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:04 INFO mapred.MapTask: bufstart = 12571307; bufend = 16762033; bufvoid = 99614720
12/10/18 22:19:04 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:19:05 INFO mapred.MapTask: Finished spill 3
12/10/18 22:19:06 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:06 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:06 INFO mapred.MapTask: bufstart = 16762033; bufend = 20952800; bufvoid = 99614720
12/10/18 22:19:06 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:19:06 INFO mapred.JobClient: map 76% reduce 0%
12/10/18 22:19:06 INFO mapred.MapTask: Finished spill 4
12/10/18 22:19:07 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:07 INFO mapred.MapTask: bufstart = 20952800; bufend = 25143412; bufvoid = 99614720
12/10/18 22:19:07 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:19:08 INFO mapred.MapTask: Finished spill 5
12/10/18 22:19:08 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:08 INFO mapred.MapTask: bufstart = 25143412; bufend = 29334124; bufvoid = 99614720
12/10/18 22:19:08 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:19:09 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:09 INFO mapred.MapTask: Finished spill 6
12/10/18 22:19:09 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:09 INFO mapred.MapTask: bufstart = 29334124; bufend = 33524778; bufvoid = 99614720
12/10/18 22:19:09 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:19:09 INFO mapred.JobClient: map 81% reduce 0%
12/10/18 22:19:10 INFO mapred.MapTask: Finished spill 7
12/10/18 22:19:10 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:10 INFO mapred.MapTask: bufstart = 33524778; bufend = 37715716; bufvoid = 99614720
12/10/18 22:19:10 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:19:11 INFO mapred.MapTask: Finished spill 8
12/10/18 22:19:11 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:11 INFO mapred.MapTask: bufstart = 37715716; bufend = 41906544; bufvoid = 99614720
12/10/18 22:19:11 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:19:12 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:12 INFO mapred.MapTask: Finished spill 9
12/10/18 22:19:12 INFO mapred.JobClient: map 88% reduce 0%
12/10/18 22:19:12 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:12 INFO mapred.MapTask: bufstart = 41906544; bufend = 46097751; bufvoid = 99614720
12/10/18 22:19:12 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:19:13 INFO mapred.MapTask: Finished spill 10
12/10/18 22:19:13 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:13 INFO mapred.MapTask: bufstart = 46097751; bufend = 50288897; bufvoid = 99614720
12/10/18 22:19:13 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:19:14 INFO mapred.MapTask: Finished spill 11
12/10/18 22:19:14 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:14 INFO mapred.MapTask: bufstart = 50288897; bufend = 54480358; bufvoid = 99614720
12/10/18 22:19:14 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:19:15 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:15 INFO mapred.MapTask: Finished spill 12
12/10/18 22:19:15 INFO mapred.JobClient: map 94% reduce 0%
12/10/18 22:19:15 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:15 INFO mapred.MapTask: bufstart = 54480358; bufend = 58671642; bufvoid = 99614720
12/10/18 22:19:15 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:19:16 INFO mapred.MapTask: Finished spill 13
12/10/18 22:19:16 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:16 INFO mapred.MapTask: bufstart = 58671642; bufend = 62863227; bufvoid = 99614720
12/10/18 22:19:16 INFO mapred.MapTask: kvstart = 65525; kvend = 327669; length = 327680
12/10/18 22:19:17 INFO mapred.MapTask: Finished spill 14
12/10/18 22:19:17 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:17 INFO mapred.MapTask: bufstart = 62863227; bufend = 67054803; bufvoid = 99614720
12/10/18 22:19:17 INFO mapred.MapTask: kvstart = 327669; kvend = 262132; length = 327680
12/10/18 22:19:17 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:19:18 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:18 INFO mapred.MapTask: Finished spill 15
12/10/18 22:19:18 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:19:19 INFO mapred.MapTask: Finished spill 16
12/10/18 22:19:19 INFO mapred.Merger: Merging 17 sorted segments
12/10/18 22:19:19 INFO mapred.Merger: Merging 8 intermediate segments out of a total of 17
12/10/18 22:19:21 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:21 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 75504227 bytes
12/10/18 22:19:24 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:27 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:27 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
12/10/18 22:19:27 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:134217728+67108864
12/10/18 22:19:27 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000002_0' done.
12/10/18 22:19:27 INFO mapred.MapTask: numReduceTasks: 1
12/10/18 22:19:27 INFO mapred.MapTask: io.sort.mb = 100
12/10/18 22:19:28 INFO mapred.MapTask: data buffer = 79691776/99614720
12/10/18 22:19:28 INFO mapred.MapTask: record buffer = 262144/327680
12/10/18 22:19:28 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:28 INFO mapred.MapTask: bufstart = 0; bufend = 4191578; bufvoid = 99614720
12/10/18 22:19:28 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
12/10/18 22:19:29 INFO mapred.MapTask: Finished spill 0
12/10/18 22:19:29 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:29 INFO mapred.MapTask: bufstart = 4191578; bufend = 8383078; bufvoid = 99614720
12/10/18 22:19:29 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
12/10/18 22:19:30 INFO mapred.MapTask: Finished spill 1
12/10/18 22:19:30 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:30 INFO mapred.MapTask: bufstart = 8383078; bufend = 12574490; bufvoid = 99614720
12/10/18 22:19:30 INFO mapred.MapTask: kvstart = 196607; kvend = 131070; length = 327680
12/10/18 22:19:31 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:31 INFO mapred.MapTask: Finished spill 2
12/10/18 22:19:31 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:31 INFO mapred.MapTask: bufstart = 12574490; bufend = 16766127; bufvoid = 99614720
12/10/18 22:19:31 INFO mapred.MapTask: kvstart = 131070; kvend = 65533; length = 327680
12/10/18 22:19:31 INFO mapred.JobClient: map 80% reduce 0%
12/10/18 22:19:32 INFO mapred.MapTask: Finished spill 3
12/10/18 22:19:32 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:32 INFO mapred.MapTask: bufstart = 16766127; bufend = 20958217; bufvoid = 99614720
12/10/18 22:19:32 INFO mapred.MapTask: kvstart = 65533; kvend = 327677; length = 327680
12/10/18 22:19:33 INFO mapred.MapTask: Finished spill 4
12/10/18 22:19:33 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:33 INFO mapred.MapTask: bufstart = 20958217; bufend = 25150065; bufvoid = 99614720
12/10/18 22:19:33 INFO mapred.MapTask: kvstart = 327677; kvend = 262140; length = 327680
12/10/18 22:19:34 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:34 INFO mapred.MapTask: Finished spill 5
12/10/18 22:19:34 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:34 INFO mapred.MapTask: bufstart = 25150065; bufend = 29341612; bufvoid = 99614720
12/10/18 22:19:34 INFO mapred.MapTask: kvstart = 262140; kvend = 196603; length = 327680
12/10/18 22:19:34 INFO mapred.JobClient: map 85% reduce 0%
12/10/18 22:19:35 INFO mapred.MapTask: Finished spill 6
12/10/18 22:19:35 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:35 INFO mapred.MapTask: bufstart = 29341612; bufend = 33533322; bufvoid = 99614720
12/10/18 22:19:35 INFO mapred.MapTask: kvstart = 196603; kvend = 131066; length = 327680
12/10/18 22:19:36 INFO mapred.MapTask: Finished spill 7
12/10/18 22:19:36 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:36 INFO mapred.MapTask: bufstart = 33533322; bufend = 37725421; bufvoid = 99614720
12/10/18 22:19:36 INFO mapred.MapTask: kvstart = 131066; kvend = 65529; length = 327680
12/10/18 22:19:37 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:37 INFO mapred.MapTask: Finished spill 8
12/10/18 22:19:37 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:37 INFO mapred.MapTask: bufstart = 37725421; bufend = 41917251; bufvoid = 99614720
12/10/18 22:19:37 INFO mapred.MapTask: kvstart = 65529; kvend = 327673; length = 327680
12/10/18 22:19:37 INFO mapred.JobClient: map 90% reduce 0%
12/10/18 22:19:38 INFO mapred.MapTask: Finished spill 9
12/10/18 22:19:38 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:38 INFO mapred.MapTask: bufstart = 41917251; bufend = 46108919; bufvoid = 99614720
12/10/18 22:19:38 INFO mapred.MapTask: kvstart = 327673; kvend = 262136; length = 327680
12/10/18 22:19:39 INFO mapred.MapTask: Finished spill 10
12/10/18 22:19:39 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:39 INFO mapred.MapTask: bufstart = 46108919; bufend = 50300567; bufvoid = 99614720
12/10/18 22:19:39 INFO mapred.MapTask: kvstart = 262136; kvend = 196599; length = 327680
12/10/18 22:19:40 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:40 INFO mapred.MapTask: Finished spill 11
12/10/18 22:19:40 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:40 INFO mapred.MapTask: bufstart = 50300567; bufend = 54492159; bufvoid = 99614720
12/10/18 22:19:40 INFO mapred.MapTask: kvstart = 196599; kvend = 131062; length = 327680
12/10/18 22:19:40 INFO mapred.JobClient: map 95% reduce 0%
12/10/18 22:19:41 INFO mapred.MapTask: Finished spill 12
12/10/18 22:19:41 INFO mapred.MapTask: Spilling map output: record full = true
12/10/18 22:19:41 INFO mapred.MapTask: bufstart = 54492159; bufend = 58684086; bufvoid = 99614720
12/10/18 22:19:41 INFO mapred.MapTask: kvstart = 131062; kvend = 65525; length = 327680
12/10/18 22:19:42 INFO mapred.MapTask: Finished spill 13
12/10/18 22:19:42 INFO mapred.MapTask: Starting flush of map output
12/10/18 22:19:43 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:43 INFO mapred.MapTask: Finished spill 14
12/10/18 22:19:43 INFO mapred.Merger: Merging 15 sorted segments
12/10/18 22:19:43 INFO mapred.Merger: Merging 6 intermediate segments out of a total of 15
12/10/18 22:19:43 INFO mapred.JobClient: map 100% reduce 0%
12/10/18 22:19:46 INFO mapred.Merger: Down to the last merge-pass, with 10 segments left of total size: 70597223 bytes
12/10/18 22:19:46 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:49 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:52 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
12/10/18 22:19:52 INFO mapred.LocalJobRunner: hdfs://localhost:9100/user/tanglg1987/input/cite75_99.txt:201326592+62748839
12/10/18 22:19:52 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000003_0' done.
12/10/18 22:19:52 INFO mapred.LocalJobRunner:
12/10/18 22:19:52 INFO mapred.Merger: Merging 4 sorted segments
12/10/18 22:19:53 INFO mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 297120317 bytes
12/10/18 22:19:53 INFO mapred.LocalJobRunner:
12/10/18 22:19:58 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:19:58 INFO mapred.JobClient: map 100% reduce 69%
12/10/18 22:20:01 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:01 INFO mapred.JobClient: map 100% reduce 72%
12/10/18 22:20:04 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:04 INFO mapred.JobClient: map 100% reduce 74%
12/10/18 22:20:07 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:07 INFO mapred.JobClient: map 100% reduce 77%
12/10/18 22:20:10 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:10 INFO mapred.JobClient: map 100% reduce 80%
12/10/18 22:20:13 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:13 INFO mapred.JobClient: map 100% reduce 82%
12/10/18 22:20:16 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:16 INFO mapred.JobClient: map 100% reduce 84%
12/10/18 22:20:19 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:19 INFO mapred.JobClient: map 100% reduce 87%
12/10/18 22:20:22 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:22 INFO mapred.JobClient: map 100% reduce 89%
12/10/18 22:20:25 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:25 INFO mapred.JobClient: map 100% reduce 91%
12/10/18 22:20:28 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:28 INFO mapred.JobClient: map 100% reduce 94%
12/10/18 22:20:31 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:31 INFO mapred.JobClient: map 100% reduce 96%
12/10/18 22:20:34 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:34 INFO mapred.JobClient: map 100% reduce 99%
12/10/18 22:20:35 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/18 22:20:35 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:35 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/18 22:20:35 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output
12/10/18 22:20:35 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:35 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/18 22:20:35 INFO mapred.JobClient: map 100% reduce 100%
12/10/18 22:20:35 INFO mapred.JobClient: Job complete: job_local_0001
12/10/18 22:20:35 INFO mapred.JobClient: Counters: 15
12/10/18 22:20:35 INFO mapred.JobClient: FileSystemCounters
12/10/18 22:20:35 INFO mapred.JobClient: FILE_BYTES_READ=1853504370
12/10/18 22:20:35 INFO mapred.JobClient: HDFS_BYTES_READ=930853207
12/10/18 22:20:35 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2603767487
12/10/18 22:20:35 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=158078539
12/10/18 22:20:35 INFO mapred.JobClient: Map-Reduce Framework
12/10/18 22:20:35 INFO mapred.JobClient: Reduce input groups=3258984
12/10/18 22:20:35 INFO mapred.JobClient: Combine output records=0
12/10/18 22:20:35 INFO mapred.JobClient: Map input records=16522439
12/10/18 22:20:35 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/18 22:20:35 INFO mapred.JobClient: Reduce output records=3258984
12/10/18 22:20:35 INFO mapred.JobClient: Spilled Records=57431615
12/10/18 22:20:35 INFO mapred.JobClient: Map output bytes=264075431
12/10/18 22:20:35 INFO mapred.JobClient: Map input bytes=264075431
12/10/18 22:20:35 INFO mapred.JobClient: Combine input records=0
12/10/18 22:20:35 INFO mapred.JobClient: Map output records=16522439
12/10/18 22:20:35 INFO mapred.JobClient: Reduce input records=16522439
.LocalJobRunner: reduce > reduce
12/10/18 22:20:07 INFO mapred.JobClient: map 100% reduce 77%
12/10/18 22:20:10 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:10 INFO mapred.JobClient: map 100% reduce 80%
12/10/18 22:20:13 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:13 INFO mapred.JobClient: map 100% reduce 82%
12/10/18 22:20:16 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:16 INFO mapred.JobClient: map 100% reduce 84%
12/10/18 22:20:19 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:19 INFO mapred.JobClient: map 100% reduce 87%
12/10/18 22:20:22 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:22 INFO mapred.JobClient: map 100% reduce 89%
12/10/18 22:20:25 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:25 INFO mapred.JobClient: map 100% reduce 91%
12/10/18 22:20:28 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:28 INFO mapred.JobClient: map 100% reduce 94%
12/10/18 22:20:31 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:31 INFO mapred.JobClient: map 100% reduce 96%
12/10/18 22:20:34 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:34 INFO mapred.JobClient: map 100% reduce 99%
12/10/18 22:20:35 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/10/18 22:20:35 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:35 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/10/18 22:20:35 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://localhost:9100/user/tanglg1987/output
12/10/18 22:20:35 INFO mapred.LocalJobRunner: reduce > reduce
12/10/18 22:20:35 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
12/10/18 22:20:35 INFO mapred.JobClient: map 100% reduce 100%
12/10/18 22:20:35 INFO mapred.JobClient: Job complete: job_local_0001
12/10/18 22:20:35 INFO mapred.JobClient: Counters: 15
12/10/18 22:20:35 INFO mapred.JobClient: FileSystemCounters
12/10/18 22:20:35 INFO mapred.JobClient: FILE_BYTES_READ=1853504370
12/10/18 22:20:35 INFO mapred.JobClient: HDFS_BYTES_READ=930853207
12/10/18 22:20:35 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2603767487
12/10/18 22:20:35 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=158078539
12/10/18 22:20:35 INFO mapred.JobClient: Map-Reduce Framework
12/10/18 22:20:35 INFO mapred.JobClient: Reduce input groups=3258984
12/10/18 22:20:35 INFO mapred.JobClient: Combine output records=0
12/10/18 22:20:35 INFO mapred.JobClient: Map input records=16522439
12/10/18 22:20:35 INFO mapred.JobClient: Reduce shuffle bytes=0
12/10/18 22:20:35 INFO mapred.JobClient: Reduce output records=3258984
12/10/18 22:20:35 INFO mapred.JobClient: Spilled Records=57431615
12/10/18 22:20:35 INFO mapred.JobClient: Map output bytes=264075431
12/10/18 22:20:35 INFO mapred.JobClient: Map input bytes=264075431
12/10/18 22:20:35 INFO mapred.JobClient: Combine input records=0
12/10/18 22:20:35 INFO mapred.JobClient: Map output records=16522439
12/10/18 22:20:35 INFO mapred.JobClient: Reduce input records=16522439
第七步：查看结果集，运行结果如下：

你可能感兴趣的:(linux,String,buffer,input,csv,output)

android系统selinux中添加新属性property 辉色投像
1.定位/android/system/sepolicy/private/property_contexts声明属性开头：persist.charge声明属性类型：u:object_r:system_prop:s0图12.定位到android/system/sepolicy/public/domain.te删除neverallow{domain-init}default_prop:property
LocalDateTime 转 String igotyback java 开发语言
importjava.time.LocalDateTime;importjava.time.format.DateTimeFormatter;publicclassMain{publicstaticvoidmain(String[]args){//获取当前时间LocalDateTimenow=LocalDateTime.now();//定义日期格式化器DateTimeFormatterformat
Linux下QT开发的动态库界面弹出操作（SDL2） 13jjyao QT类 qt 开发语言 sdl2 linux
需求：操作系统为linux，开发框架为qt，做成需带界面的qt动态库，调用方为java等非qt程序难点：调用方为java等非qt程序，也就是说调用方肯定不带QApplication::exec()，缺少了这个，QTimer等事件和QT创建的窗口将不能弹出(包括opencv也是不能弹出)；这与qt调用本身qt库是有本质的区别的思路：1.调用方缺QApplication::exec()，那么我们在接口
每日一题——第九十题互联网打工人no1 C语言程序设计每日一练 c语言
题目：判断子串是否与主串匹配#include#include#include//////判断子串是否在主串中匹配//////主串///子串///boolisSubstring(constchar*str,constchar*substr){intlenstr=strlen(str);//计算主串的长度intlenSub=strlen(substr);//计算子串的长度//遍历主字符串，对每个可能得
每日一题——第八十四题互联网打工人no1 C语言程序设计每日一练 c语言
题目：编写函数1、输入10个职工的姓名和职工号2、按照职工由大到小顺序排列，姓名顺序也随之调整3、要求输入一个职工号，用折半查找法找出该职工的姓名#define_CRT_SECURE_NO_WARNINGS#include#include#defineMAX_EMPLOYEES10typedefstruct{intid;charname[50];}Empolyee;voidinputEmploye
每日一题——第八十二题互联网打工人no1 C语言程序设计每日一练 c语言
题目：将一个控制台输入的字符串中的所有元音字母复制到另一字符串中#include#include#include#include#defineMAX_INPUT1024boolisVowel(charp);intmain(){charinput[MAX_INPUT];charoutput[MAX_INPUT];printf("请输入一串字符串：\n");fgets(input,sizeof(inp
每日一题——第八十三题互联网打工人no1 C语言程序设计每日一练 c语言
题目：将输入的整形数字输出,输出1990，输出"1990"#include#defineMAX_INPUT1024intmain(){intarrr_num[MAX_INPUT];intnum,i=0;printf("请输入一个数字：");scanf_s("%d",&num);while(num!=0){arrr_num[i++]=num%10;num/=10;}printf("\"");for(
C#中使用split分割字符串互联网打工人no1 c#
1、用字符串分隔：usingSystem.Text.RegularExpressions;stringstr="aaajsbbbjsccc";string[]sArray=Regex.Split(str,"js",RegexOptions.IgnoreCase);foreach(stringiinsArray)Response.Write(i.ToString()+"");输出结果：aaabbbc
linux sdl windows.h,Windows下的SDL安装奔跑吧linux内核 linux sdl windows.h
首先你要下载并安装SDL开发包。如果装在C盘下，路径为C:\SDL1.2.5如果在WINDOWS下。你可以按以下步骤：1.打开VC++，点击"Tools",Options2,点击directories选项3.选择"Includefiles"增加一个新的路径。"C:\SDL1.2.5\include"4，现在选择"Libaryfiles“增加"C:\SDL1.2.5\lib"现在你可以开始编写你的第
linux中sdl的使用教程,sdl使用入门 Melissa Corvinus linux中sdl的使用教程
本文通过一个简单示例讲解SDL的基本使用流程。示例中展示一个窗口，窗口里面有个随机颜色快随机移动。当我们鼠标点击关闭按钮时间窗口关闭。基本步骤如下：1.初始化SDL并创建一个窗口。SDL_Init()初始化SDL_CreateWindow()创建窗口2.纹理渲染存储RGB和存储纹理的区别：比如一个从左到右由红色渐变到蓝色的矩形，用存储RGB的话就需要把矩形中每个点的具体颜色值存储下来；而纹理只是一
PHP环境搭建详细教程好看资源平台前端 php
PHP是一个流行的服务器端脚本语言，广泛用于Web开发。为了使PHP能够在本地或服务器上运行，我们需要搭建一个合适的PHP环境。本教程将结合最新资料，介绍在不同操作系统上搭建PHP开发环境的多种方法，包括Windows、macOS和Linux系统的安装步骤，以及本地和Docker环境的配置。1.PHP环境搭建概述PHP环境的搭建主要分为以下几类：集成开发环境：例如XAMPP、WAMP、MAMP，这
使用 FinalShell 进行远程连接（ssh 远程连接 Linux 服务器）编程经验分享开发工具服务器 ssh linux
目录前言基本使用教程新建远程连接连接主机自定义命令路由追踪前言后端开发，必然需要和服务器打交道，部署应用，排查问题，查看运行日志等等。一般服务器都是集中部署在机房中，也有一些直接是云服务器，总而言之，程序员不可能直接和服务器直接操作，一般都是通过ssh连接来登录服务器。刚接触远程连接时，使用的是XSHELL来远程连接服务器，连接上就能够操作远程服务器了，但是仅用XSHELL并没有上传下载文件的功能
python是什么意思中文-在python中%是什么意思编程大乐趣
Python中%有两种：1、数值运算：%代表取模，返回除法的余数。如：>>>7%212、%操作符（字符串格式化，stringformatting），说明如下：%[(name)][flags][width].[precision]typecode(name)为命名flags可以有+，-，''或0。+表示右对齐。-表示左对齐。''为一个空格，表示在正数的左侧填充一个空格，从而与负数对齐。0表示使用0填
libyuv之linux编译 jaronho Linux linux 运维服务器
文章目录一、下载源码二、编译源码三、注意事项1、银河麒麟系统（aarch64）（1）解决armv8-a+dotprod+i8mm指令集支持问题（2）解决armv9-a+sve2指令集支持问题一、下载源码到GitHub网站下载https://github.com/lemenkov/libyuv源码，或者用直接用git克隆到本地，如：gitclonehttps://github.com/lemenko
python os 环境变量 CV矿工 python 开发语言 numpy
环境变量：环境变量是程序和操作系统之间的通信方式。有些字符不宜明文写进代码里，比如数据库密码，个人账户密码，如果写进自己本机的环境变量里，程序用的时候通过os.environ.get（）取出来就行了。os.environ是一个环境变量的字典。环境变量的相关操作importos"""设置/修改环境变量：os.environ[‘环境变量名称’]=‘环境变量值’#其中key和value均为string类
Redis系列：Geo 类型赋能亿级地图位置计算 Ly768768 redis bootstrap 数据库
1前言我们在篇深刻理解高性能Redis的本质的时候就介绍过Redis的几种基本数据结构，它是基于不同业务场景而设计的：动态字符串(REDIS_STRING)：整数(REDIS_ENCODING_INT)、字符串(REDIS_ENCODING_RAW)双端列表(REDIS_ENCODING_LINKEDLIST)压缩列表(REDIS_ENCODING_ZIPLIST)跳跃表(REDIS_ENCODI
ARM驱动学习之5 LEDS驱动 JT灬新一嵌入式 C 底层 arm开发学习单片机
ARM驱动学习之5LEDS驱动知识点：•linuxGPIO申请函数和赋值函数–gpio_request–gpio_set_value•三星平台配置GPIO函数–s3c_gpio_cfgpin•GPIO配置输出模式的宏变量–S3C_GPIO_OUTPUT注意点：DRIVER_NAME和DEVICE_NAME匹配。实现步骤：1.加入需要的头文件：//Linux平台的gpio头文件#include//三
C++ | Leetcode C++题解之第409题最长回文串 Ddddddd_158 经验分享 C++Leetcode 题解
题目：题解：classSolution{public:intlongestPalindrome(strings){unordered_mapcount;intans=0;for(charc:s)++count[c];for(autop:count){intv=p.second;ans+=v/2*2;if(v%2==1andans%2==0)++ans;}returnans;}};
【华为OD技术面试真题精选 - 非技术题】 -HR面，综合面_华为od hr面一个射手座的程序媛程序员华为od 面试职场和发展
最后的话最近很多小伙伴找我要Linux学习资料，于是我翻箱倒柜，整理了一些优质资源，涵盖视频、电子书、PPT等共享给大家！资料预览给大家整理的视频资料：给大家整理的电子书资料：如果本文对你有帮助，欢迎点赞、收藏、转发给朋友，让我有持续创作的动力！网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。需要这份系统化的资料的朋友，可以点击这里获
简介Shell、zsh、bash zhaosuningsn Shell zsh bash shell linux bash
Shell是Linux和Unix的外壳，类似衣服，负责外界与Linux和Unix内核的交互联系。例如接收终端用户及各种应用程序的命令，把接收的命令翻译成内核能理解的语言，传递给内核，并把内核处理接收的命令的结果返回给外界，即Shell是外界和内核沟通的桥梁或大门。Linux和Unix提供了多种Shell，其中有种bash，当然还有其他好多种。Mac电脑中不但有bash，还有一个zsh，预装的，据说
Linux MariaDB使用OpenSSL安装SSL证书 Meta39 MySQL Oracle MariaDB Linux Windows ssl linux mariadb
进入到证书存放目录，批量删除.pem证书警告：确保已经进入到证书存放目录find.-typef-iname\*.pem-delete查看是否安装OpenSSLopensslversion没有则安装yuminstallopensslopenssl-devel开启SSL编辑/etc/my.cnf文件（没有的话就创建，但是要注意，在/etc/my.cnf.d/server.cnf配置了datadir的，
【从浅识到熟知Linux】Linux发展史 Jammingpro 从浅学到熟知Linux linux 运维服务器
归属专栏：从浅学到熟知Linux个人主页：Jammingpro每日努力一点点，技术变化看得见文章前言：本篇文章记录Linux发展的历史，因在介绍Linux过程中涉及的其他操作系统及人物，本文对相关内容也有所介绍。文章目录Unix发展史Linux发展史开源Linux官网企业应用情况发行版本在学习Linux前，我们可能都会问Linux从哪里来？它是如何发展的。但在介绍Linux之前，需要先介绍一下Un
linux 发展史种树的猴子内核 java 操作系统 linux 大数据
linux发展史说明此前对linux认识模糊一知半解，近期通过学习将自己对于linux的发展总结一下方便大家日后的学习。那Linux是目前一款非常火热的开源操作系统，可是linux是什么时候出现的，又是因为什么样的原因被开发出来的呢。以下将对linux的发展历程进行详细的讲解。目录一、Linux发展背景二、UINIX的诞生三、UNIX的重要分支-BSD的诞生四、Minix的诞生五、GNU与Free
Linux sh命令 fengyehongWorld Linux linux
目录一.基本语法二.选项2.1-c字符串中读取内容，并执行2.1.1基本用法2.1.2获取当前目录下失效的超链接2.2-x每个命令执行之前，将其打印出来2.3结合Here文档使用一.基本语法⏹Linux和Unix系统中用于执行shell脚本或运行命令的命令。sh[选项][脚本文件][参数...]⏹选项-c：从字符串中读取内容，并执行。-x：在每个命令执行之前，将其打印出来。-s：从标准流中读取内容
推荐算法_隐语义-梯度下降 _feivirus_ 算法机器学习和数学推荐算法机器学习隐语义
importnumpyasnp1.模型实现"""inputrate_matrix:M行N列的评分矩阵，值为P*Q.P:初始化用户特征矩阵M*K.Q:初始化物品特征矩阵K*N.latent_feature_cnt:隐特征的向量个数max_iteration:最大迭代次数alpha:步长lamda:正则化系数output分解之后的P和Q"""defLFM_grad_desc(rate_matrix,l
Linux vi常用命令 fengyehongWorld Linux linux
参考资料viコマンド（vimコマンド）リファレンス目录一.保存系命令二.删除系命令三.移动系命令四.复制粘贴系命令一.保存系命令⏹保存并退出:wq⏹强制保存并退出:wq!⏹退出(文件未编辑):q⏹强制退出(忽略已编辑内容):q!⏹另存为:w新文件名二.删除系命令⏹删除当前行dd⏹清空整个文档gg：移动到文档顶部dG：删除到最后一行ggdG三.移动系命令⏹移动到文档顶部gg⏹移动到文档底部#方式1G
用Python实现简单的猜数字游戏程序媛了了 python 游戏 java
猜数字游戏代码：importrandomdefpythonit():a=random.randint(1,100)n=int(input("输入你猜想的数字："))whilen!=a:ifn>a:print("很遗憾，猜大了")n=int(input("请再次输入你猜想的数字："))elifna::如果玩家猜的数字n大于随机数字a，则输出"很遗憾，猜大了"，并提示玩家再次输入。elifn
2024.9.6 Python，华为笔试题总结，字符串格式化，字符串操作，广度优先搜索解决公司组织绩效互评问题，无向图 RaidenQ python 华为 leetcode 算法力扣广度优先无向图
1.字符串格式化name="Alice"age=30formatted_string="Name:{},Age:{}".format(name,age)print(formatted_string)或者name="Alice"age=30formatted_string=f"Name:{name},Age:{age}"print(formatted_string)2.网络健康检查第一行有两个整数m
Linux查看服务器日志 TPBoreas 运维 linux 运维
一、tail这个是我最常用的一种查看方式用法如下：tail-n10test.log查询日志尾部最后10行的日志;tail-n+10test.log查询10行之后的所有日志;tail-fn10test.log循环实时查看最后1000行记录(最常用的)一般还会配合着grep用，(实时抓包)例如:tail-fn1000test.log|grep'关键字'（动态抓包）tail-fn1000test.log
python怎么将png转为tif_png转tif weixin_39977276
发国外的文章要求图片是tif，cmyk色彩空间的。大小尺寸还有要求。比如网上大神多，找到了一段代码，感谢！https://www.jianshu.com/p/ec2af4311f56https://github.com/KevinZc007/image2Tifimportjava.awt.image.BufferedImage;importjava.io.File;importjava.io.Fi
java责任链模式 3213213333332132 java 责任链模式村民告县长
责任链模式，通常就是一个请求从最低级开始往上层层的请求，当在某一层满足条件时，请求将被处理，当请求到最高层仍未满足时，则请求不会被处理。就是一个请求在这个链条的责任范围内，会被相应的处理，如果超出链条的责任范围外，请求不会被相应的处理。下面代码模拟这样的效果：创建一个政府抽象类,方便所有的具体政府部门继承它。 package 责任链模式; /** *
linux、mysql、nginx、tomcat 性能参数优化 ronin47
一、linux 系统内核参数 /etc/sysctl.conf文件常用参数 net.core.netdev_max_backlog = 32768 #允许送到队列的数据包的最大数目 net.core.rmem_max = 8388608 #SOCKET读缓存区大小 net.core.wmem_max = 8388608 #SOCKET写缓存区大
php命令行界面 dcj3sjt126com PHP cli
常用选项 php -v php -i PHP安装的有关信息 php -h 访问帮助文件 php -m 列出编译到当前PHP安装的所有模块执行一段代码 php -r 'echo "hello, world!";' php -r 'echo "Hello, World!\n";' php -r '$ts = filemtime("
Filter&Session 171815164 session
Filter HttpServletRequest requ = (HttpServletRequest) req; HttpSession session = requ.getSession(); if (session.getAttribute("admin") == null) { PrintWriter out = res.ge
连接池与Spring,Hibernate结合 g21121 Hibernate
前几篇关于Java连接池的介绍都是基于Java应用的，而我们常用的场景是与Spring和ORM框架结合，下面就利用实例学习一下这方面的配置。 1.下载相关内容： &nb
[简单]mybatis判断数字类型 53873039oycg mybatis
昨天同事反馈mybatis保存不了int类型的属性,一直报错，错误信息如下: Caused by: java.lang.NumberFormatException: For input string: "null" at sun.mis
项目启动时或者启动后ava.lang.OutOfMemoryError: PermGen space 程序员是怎么炼成的 eclipse jvm tomcat catalina.sh eclipse.ini
在启动比较大的项目时，因为存在大量的jsp页面，所以在编译的时候会生成很多的.class文件，.class文件是都会被加载到jvm的方法区中，如果要加载的class文件很多，就会出现方法区溢出异常 java.lang.OutOfMemoryError: PermGen space. 解决办法是点击eclipse里的tomcat，在
我的crm小结 aijuans crm
各种原因吧，crm今天才完了。主要是接触了几个新技术： Struts2、poi、ibatis这几个都是以前的项目中用过的。 Jsf、tapestry是这次新接触的，都是界面层的框架，用起来也不难。思路和struts不太一样，传说比较简单方便。不过个人感觉还是struts用着顺手啊，当然springmvc也很顺手，不知道是因为习惯还是什么。jsf和tapestry应用的时候需要知道他们的标签、主
spring里配置使用hibernate的二级缓存几步 antonyup_2006 java spring Hibernate xml cache
．在spring的配置文件中 applicationContent.xml，hibernate部分加入 xml 代码 <prop key="hibernate.cache.provider_class">org.hibernate.cache.EhCacheProvider</prop> <prop key="hi
JAVA基础面试题百合不是茶抽象实现接口 String类接口继承抽象类继承实体类自定义异常
/* * 栈（stack）：主要保存基本类型（或者叫内置类型）（char、byte、short、 *int、long、 float、double、boolean）和对象的引用，数据可以共享，速度仅次于 * 寄存器（register），快于堆。堆（heap）：用于存储对象。 */ &
让sqlmap文件 "继承" 起来 bijian1013 java ibatis sqlmap
多个项目中使用ibatis , 和数据库表对应的 sqlmap文件（增删改查等基本语句)，dao, pojo 都是由工具自动生成的, 现在将这些自动生成的文件放在一个单独的工程中，其它项目工程中通过jar包来引用，并通过"继承"为基础的sqlmap文件，dao,pojo 添加新的方法来满足项
精通Oracle10编程SQL(13)开发触发器 bijian1013 oracle 数据库 plsql
/* *开发触发器 */ --得到日期是周几 select to_char(sysdate+4,'DY','nls_date_language=AMERICAN') from dual; select to_char(sysdate,'DY','nls_date_language=AMERICAN') from dual; --建立BEFORE语句触发器 CREATE O
【EhCache三】EhCache查询 bit1129 ehcache
本文介绍EhCache查询缓存中数据，EhCache提供了类似Hibernate的查询API，可以按照给定的条件进行查询。要对EhCache进行查询，需要在ehcache.xml中设定要查询的属性数据准备 @Before public void setUp() { //加载EhCache配置文件 Inpu
CXF框架入门实例白糖_ spring Web 框架 webservice servlet
CXF是apache旗下的开源框架，由Celtix + XFire这两门经典的框架合成，是一套非常流行的web service框架。它提供了JAX-WS的全面支持，并且可以根据实际项目的需要，采用代码优先（Code First）或者 WSDL 优先（WSDL First）来轻松地实现 Web Services 的发布和使用，同时它能与spring进行完美结合。在apache cxf官网提供
angular.equals boyitech AngularJS AngularJS API AnguarJS 中文API angular.equals
angular.equals 描述: 比较两个值或者两个对象是不是相等。还支持值的类型，正则表达式和数组的比较。两个值或对象被认为是相等的前提条件是以下的情况至少能满足一项：两个值或者对象能通过=== （恒等）的比较两个值或者对象是同样类型，并且他们的属性都能通过angular
java-腾讯暑期实习生-输入一个数组A[1,2,...n]，求输入B，使得数组B中的第i个数字B[i]=A[0]*A[1]*...*A[i-1]*A[i+1] bylijinnan java
这道题的具体思路请参看何海涛的微博：http://weibo.com/zhedahht import java.math.BigInteger; import java.util.Arrays; public class CreateBFromATencent { /** * 题目：输入一个数组A[1,2,...n]，求输入B，使得数组B中的第i个数字B[i]=A
FastDFS 的安装和配置修订版 Chen.H linux fastDFS 分布式文件系统
FastDFS Home:http://code.google.com/p/fastdfs/ 1. 安装 http://code.google.com/p/fastdfs/wiki/Setup http://hi.baidu.com/leolance/blog/item/3c273327978ae55f93580703.html 安装libevent (对libevent的版本要求为1.4.
[强人工智能]拓扑扫描与自适应构造器 comsci 人工智能
当我们面对一个有限拓扑网络的时候,在对已知的拓扑结构进行分析之后,发现在连通点之后,还存在若干个子网络,且这些网络的结构是未知的,数据库中并未存在这些网络的拓扑结构数据....这个时候,我们该怎么办呢? 那么,现在我们必须设计新的模块和代码包来处理上面的问题
oracle merge into的用法 daizj oracle sql merget into
Oracle中merge into的使用 http://blog.csdn.net/yuzhic/article/details/1896878 http://blog.csdn.net/macle2010/article/details/5980965 该命令使用一条语句从一个或者多个数据源中完成对表的更新和插入数据. ORACLE 9i 中，使用此命令必须同时指定UPDATE 和INSE
不适合使用Hadoop的场景 datamachine hadoop
转自：http://dev.yesky.com/296/35381296.shtml。　　Hadoop通常被认定是能够帮助你解决所有问题的唯一方案。当人们提到“大数据”或是“数据分析”等相关问题的时候，会听到脱口而出的回答：Hadoop! 实际上Hadoop被设计和建造出来，是用来解决一系列特定问题的。对某些问题来说，Hadoop至多算是一个不好的选择，对另一些问题来说，选择Ha
YII findAll的用法 dcj3sjt126com yii
看文档比较糊涂，其实挺简单的： $predictions=Prediction::model()->findAll("uid=:uid",array(":uid"=>10)); 第一个参数是选择条件：”uid=10″。其中:uid是一个占位符，在后面的array(“:uid”=>10)对齐进行了赋值；更完善的查询需要
vim 常用 NERDTree 快捷键 dcj3sjt126com vim
下面给大家整理了一些vim NERDTree的常用快捷键了，这里几乎包括了所有的快捷键了，希望文章对各位会带来帮助。切换工作台和目录 ctrl + w + h 光标 focus 左侧树形目录ctrl + w + l 光标 focus 右侧文件显示窗口ctrl + w + w 光标自动在左右侧窗口切换ctrl + w + r 移动当前窗口的布局位置 o 在已有窗口中打开文件、目录或书签，并跳
Java把目录下的文件打印出来蕃薯耀列出目录下的文件文件夹下面的文件目录下的文件
Java把目录下的文件打印出来 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 蕃薯耀 2015年7月11日 11:02:
linux远程桌面----VNCServer与rdesktop hanqunfeng Desktop
windows远程桌面到linux，需要在linux上安装vncserver，并开启vnc服务，同时需要在windows下使用vnc-viewer访问Linux。vncserver同时支持linux远程桌面到linux。 linux远程桌面到windows，需要在linux上安装rdesktop，同时开启windows的远程桌面访问。下面分别介绍，以windo
guava中的join和split功能 jackyrong java
guava库中，包含了很好的join和split的功能，例子如下： 1）将LIST转换为使用字符串连接的字符串 List<String> names = Lists.newArrayList("John", "Jane", "Adam", "Tom");
Web开发技术十年发展历程 lampcy android Web 浏览器 html5
回顾web开发技术这十年发展历程： Ajax 03年的时候我上六年级，那时候网吧刚在小县城的角落萌生。传奇，大话西游第一代网游一时风靡。我抱着试一试的心态给了网吧老板两块钱想申请个号玩玩，然后接下来的一个小时我一直在，注，册，账，号。彼时网吧用的512k的带宽，注册的时候，填了一堆信息，提交，页面跳转，嘣，”您填写的信息有误，请重填”。然后跳转回注册页面，以此循环。我现在时常想，如果当时a
架构师之mima-----------------mina的非NIO控制IOBuffer(说得比较好) nannan408 buffer
1.前言。如题。 2.代码。 IoService IoService是一个接口，有两种实现：IoAcceptor和IoConnector；其中IoAcceptor是针对Server端的实现，IoConnector是针对Client端的实现；IoService的职责包括： 1、监听器管理 2、IoHandler 3、IoSession
ORA-00054:resource busy and acquire with NOWAIT specified Everyday都不同 oracle session Lock
[Oracle] 今天对一个数据量很大的表进行操作时，出现如题所示的异常。此时表明数据库的事务处于“忙”的状态，而且被lock了，所以必须先关闭占用的session。 step1，查看被lock的session： select t2.username, t2.sid, t2.serial#, t2.logon_time from v$locked_obj
javascript学习笔记 tntxia JavaScript
javascript里面有6种基本类型的值:number、string、boolean、object、function和undefined。number：就是数字值，包括整数、小数、NaN、正负无穷。string:字符串类型、单双引号引起来的内容。boolean:true、false object:表示所有的javascript对象，不用多说function:我们熟悉的方法，也就是
Java enum的用法详解 xieke90 enum 枚举
Java中枚举实现的分析：示例： public static enum SEVERITY{ INFO,WARN,ERROR } enum很像特殊的class，实际上enum声明定义的类型就是一个类。而这些类都是类库中Enum类的子类 (java.l