Hadoop for .NET开发方式将数据加载到HDFS

Hadoop for .NET开发方式将数据加载到HDFS

参考文章：https://blog.csdn.net/WuLex/article/details/78069478

启动Visual Studio并创建一个新的C＃控制台应用程序

使用NuGet软件包管理器，
将Microsoft.Hadoop.WebClient软件包的Microsoft .NET API添加到您的项目中
包括相关的依赖项DLL

<packages>
  <package id="Microsoft.AspNet.WebApi.Client" version="4.0.20505.0" targetFramework="net452" />
  <package id="Microsoft.Data.Edm" version="5.2.0" targetFramework="net452" />
  <package id="Microsoft.Data.OData" version="5.2.0" targetFramework="net452" />
  <package id="Microsoft.Hadoop.WebClient" version="0.12.5126.42915" targetFramework="net452" />
  <package id="Microsoft.Net.Http" version="2.0.20505.0" targetFramework="net452" />
  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="1.8.0.0" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="4.5.11" targetFramework="net452" />
  <package id="System.Spatial" version="5.2.0" targetFramework="net452" />
  <package id="WindowsAzure.Storage" version="2.0.4.1" targetFramework="net452" />
packages>

请打开Program.cs文件并添加以下指令：
using Microsoft.Hadoop.WebHDFS;

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Hadoop.WebHDFS;

namespace hadoopNetDevConsole
{
    class Program
    {
        static void Main(string[] args)
        {
            string srcPath = @"D:\vm\json\test_fenfu2000_1w.json";
            string destPath = "/arcgis/test_fenfu2000_1w_9.json";
            string destDir = "/arcgis/";

            //连接到hadoop集群
            Uri myUri = new Uri("http://192.168.10.200:50070");
            string userName = "hadoop3";


            //由于把hdfs-site.xml配置文件中dfs.permissions属性 设置false
            WebHDFSClient myClient = new WebHDFSClient(myUri, "");

            //将文件加载到目标目录         
            myClient.CreateFile(srcPath, destPath).Wait();

            //列出目标目录的文件内容
            Console.WriteLine();
            Console.WriteLine("Contents of " + destDir);
            //myClient.GetDirectoryStatus(destDir).ContinueWith(
            //    ds => ds.Result.Files.ToList().ForEach(
            //    f => Console.WriteLine("t" + f.PathSuffix)
            //    ));

            Console.ReadLine();
        }
    }
}

查看上传到hdfs上的情况：
http://192.168.10.201:50070/explorer.html#/arcgis

Browse Directory

/arcgis

Permission  Owner   Group   Size    Last Modified   Replication Block Size  Name
-rwxr-xr-x  hadoop3 supergroup  9.99 MB 2018/8/17 下午8:09:35 1   128 MB  11.json
-rwxr-xr-x  dr.who  supergroup  9.99 MB 2018/8/17 下午7:46:06 3   128 MB  test_fenfu2000_1w.json
-rwxr-xr-x  hadoop3 supergroup  9.99 MB 2018/8/17 下午8:00:30 1   128 MB  test_fenfu2000_1w11.json
-rwxr-xr-x  hadoop3 supergroup  9.99 MB 2018/8/17 下午7:48:02 1   128 MB  test_fenfu2000_1w_3.json
-rwxr-xr-x  dr.who  supergroup  9.99 MB 2018/8/17 下午7:56:16 3   128 MB  test_fenfu2000_1w_9.json
-rwxr-xr-x  dr.who  supergroup  9.99 MB 2018/8/17 下午8:15:59 3   128 MB  test_fenfu2000_1w_99.json
Hadoop, 2018.

遇见报错的问题: myClient.CreateFile(srcPath, destPath).Wait();
不能解析slave2 主机名

原因是hadoop集群有三台
192.168.10.200 master
192.168.10.201 slave1
192.168.10.202 slave2

hosts c:\windows\System32\drivers\etc
添加下面的内容就OK：机器名与IP互转换的问题

0.0.0.0 account.jetbrains.com
192.168.10.200  master
192.168.10.201  slave1
192.168.10.202  slave2