Win8下安装配置Docker ToolBox并制作镜像配置Handoop集群

1、安装Docker ToolBox

      参考1：https://www.cnblogs.com/weschen/p/6697926.html

      Oracle   VirtualBox、Git和Kitematic可选，若没有玩过微软自带的Hyper-V，则不会有相应设置

2、Docker ToolBox优化配置

   2.1 设置ali镜像：

       运行Git Bash，进入终端

       xxxxMINGW64~

       $ docker-machine ssh default

       $ sudo sed -i "s|EXTRA_ARGS='|EXTRA_ARGS='--registry-mirror=【此处为你的加速地址】 |g" /var/lib/boot2docker/profile

       $ exit

       $ docker-machine restart default

       重启VirtualBox虚拟机

       启动Docker Quickstart Terminal

  2.2 更换虚拟硬盘位置

      打开Oracle VM VirtualBox，全局工具->释放(虚拟机已启动的情况下不能移动)/移动（虚拟机停止时可移动），再到虚拟机设置里：存储->添加存储附件->添加虚拟硬盘

3 拉镜像，起容器

   参考2：https://blog.csdn.net/yangym2002/article/details/79000241

4 安装配置hadoop集群

   从裸镜像安装配置环境参考3：https://blog.51cto.com/865516915/2150651

   该方法充分利用了镜像的封装可重用的特点

   直接拉Hadoop镜像安装配置hadoop集群参考4：https://www.jianshu.com/p/30604c820e9d

5 避坑提示

  5.1 容器报错，加上 -itd

    docker run --name hadoop2--add-host  ..................... 加上-itd参数启动容器后再使用docker attach进入容器

   反正hadoop需要使用ssh，最好使用 /usr/sbin/sshd -D 启动容器，再是用ssh client工具进入容器进行管理

   若使用/bin/bash起容器，则使用docker attach进入容器后需要手动启动sshd

   5.2 从零安装后文件夹瘦身

     使用Dockerfile制作的镜像1.59G，纯手工安装制作镜像1.04G

         将参考3中的Dockerfile指令手动执行，handoop安装配置好后只需制作一个镜像

     hadoop里面的share/doc目录占用空间较大，可删除

     使用jdk而不是jre，jdk里面的两个src包可以删除掉以减少空间

     java环境变量和hadoop环境变量要放到.bashrc里面去

   5.3 执行测试报关于maxiummem相关的错

         给虚拟机内存分配大点

         yarn配置调优参考5：https://blog.csdn.net/stark_summer/article/details/48494391

   5.4 Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

         参考6：https://www.cnblogs.com/tele-share/p/9497613.html

6  测试成功

[root@966316f84c04 mapreduce]# hadoop jar hadoop-mapreduce-examples-3.2.0.jar wordcount /input /output1
2019-06-24 09:49:01,834 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/172.17.0.2:8032
2019-06-24 09:49:03,201 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1561369728889_0001
2019-06-24 09:49:04,593 INFO input.FileInputFormat: Total input files to process : 1
2019-06-24 09:49:05,098 INFO mapreduce.JobSubmitter: number of splits:1
2019-06-24 09:49:05,216 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2019-06-24 09:49:05,557 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1561369728889_0001
2019-06-24 09:49:05,560 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-06-24 09:49:05,864 INFO conf.Configuration: resource-types.xml not found
2019-06-24 09:49:05,865 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-06-24 09:49:06,628 INFO impl.YarnClientImpl: Submitted application application_1561369728889_0001
2019-06-24 09:49:06,729 INFO mapreduce.Job: The url to track the job: http://hadoop-master:8088/proxy/application_1561369728889_0001/
2019-06-24 09:49:06,731 INFO mapreduce.Job: Running job: job_1561369728889_0001
2019-06-24 09:49:20,084 INFO mapreduce.Job: Job job_1561369728889_0001 running in uber mode : false
2019-06-24 09:49:20,092 INFO mapreduce.Job:  map 0% reduce 0%
2019-06-24 09:49:30,281 INFO mapreduce.Job:  map 100% reduce 0%
2019-06-24 09:49:37,356 INFO mapreduce.Job:  map 100% reduce 100%
2019-06-24 09:49:38,385 INFO mapreduce.Job: Job job_1561369728889_0001 completed successfully
2019-06-24 09:49:38,525 INFO mapreduce.Job: Counters: 54
        File System Counters
                FILE: Number of bytes read=1831
                FILE: Number of bytes written=447569
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=1468
                HDFS: Number of bytes written=1301
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
                HDFS: Number of bytes read erasure-coded=0
        Job Counters
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=6141
                Total time spent by all reduces in occupied slots (ms)=4617
                Total time spent by all map tasks (ms)=6141
                Total time spent by all reduce tasks (ms)=4617
                Total vcore-milliseconds taken by all map tasks=6141
                Total vcore-milliseconds taken by all reduce tasks=4617
                Total megabyte-milliseconds taken by all map tasks=6288384
                Total megabyte-milliseconds taken by all reduce tasks=4727808
        Map-Reduce Framework
                Map input records=31
                Map output records=179
                Map output bytes=2050
                Map output materialized bytes=1831
                Input split bytes=107
                Combine input records=179
                Combine output records=131
                Reduce input groups=131
                Reduce shuffle bytes=1831
                Reduce input records=131
                Reduce output records=131
                Spilled Records=262
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=148
                CPU time spent (ms)=1120
                Physical memory (bytes) snapshot=324399104
                Virtual memory (bytes) snapshot=5145825280
                Total committed heap usage (bytes)=170004480
                Peak Map Physical memory (bytes)=211697664
                Peak Map Virtual memory (bytes)=2569519104
                Peak Reduce Physical memory (bytes)=112701440
                Peak Reduce Virtual memory (bytes)=2576306176
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1361
        File Output Format Counters
                Bytes Written=1301