通过Java代码对HDFS进行写入与查看操作

这篇来介绍一下往HDFS写数据与查看文件信息的操作.

1.读取本地文件,并写入至HDFS.

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;

import java.io.*;
import java.net.URI;

public class CopyToHdfs {
    public static void main(String[] args) throws Exception{
        String hdfsPath = "hdfs://192.168.133.153:9000/test/write/hdfs-site.xml";//hdfs位置
        String localPath = "E:\\Hadoop\\hdfs-site.xml";//本地磁盘位置

        Configuration configuration = new Configuration();
        FileSystem fs = FileSystem.get(URI.create(hdfsPath), configuration);
        FSDataOutputStream out = fs.create(new Path(hdfsPath));//创建一个输出流
        InputStream in = new FileInputStream(new File(localPath));//从本地读取文件
        IOUtils.copyBytes(in, out, 100, true);
        System.out.println("上传完毕");
    }
}

之后使用在虚拟机使用 hdfs dfs -ls /test/write 命令查看,可以发现已经写入的文件:

[scott@master hadoop]$ hdfs dfs -ls /test/write
Found 1 items
-rw-r--r--   3 scott supergroup       2275 2019-01-20 13:11 /test/write/hdfs-site.xml

2.利用getFileStatus()展示目录信息

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;

import java.net.URI;

public class ShowFileStatus {
    public static void main(String[] args) throws Exception{
        String hdfsPath = "hdfs://192.168.133.153:9000/test/";
        FileSystem fs = FileSystem.get(URI.create(hdfsPath), new Configuration());
        Path path = new Path(hdfsPath);
        FileStatus status = fs.getFileStatus(path);
        System.out.println(status);
    }
}

输出结果:

FileStatus{
path=hdfs://192.168.133.153:9000/test; 
isDirectory=true; 
modification_time=1547954859269; 
access_time=0; 
owner=scott; 
group=supergroup; 
permission=rwxr-xr-x; 
isSymlink=false
}

这里做了一点处理,实际的输出结果这些信息全部都挤在一行.可以发现,展示的信息还是比较全面的:是否是目录.修改时间,所有者,所在组,权限,是否是符号链接等.

3.利用listStatus()显示文件夹内都有什么

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;

import java.net.URI;

public class ShowSubDirs {
    public static void main(String[] args) throws Exception{
        String dir1 = "hdfs://192.168.133.153:9000/test";
        FileSystem fs = FileSystem.get(new URI(dir1), new Configuration());
        Path path = new Path(dir1);
        FileStatus[] status = fs.listStatus(path);
        Path[] paths = FileUtil.stat2Paths(status);//又一个工具类
        for (Path p: paths) {
            System.out.println(p.toString());
        }
    }
}

输出结果:

hdfs://192.168.133.153:9000/test/read
hdfs://192.168.133.153:9000/test/write

Hadoop提供了非常实用的工具类,多多使用它们会让程序变得简单一些.