【HDFS API编程】从本地拷贝文件，从本地拷贝大文件，拷贝HDFS文件到本地
接着之前继续API操作的学习
CopyFromLocalFile: 顾名思义，从本地文件拷贝
/** * 使用Java API操作HDFS文件系统 * 关键点: * 1)create Configuration * 2)get FileSystem * 3)...It's your HDFS API operation. */ public class HDFSApp { public static final String HDFS_PATH = "hdfs://hadoop000:8020"; FileSystem fileSystem = null; Configuration configuration = null; @Before public void setUp() throws Exception{ System.out.println("setUp-----------"); configuration = new Configuration(); configuration.set("dfs.replication","1"); /* * 构造一个访问制定HDFS系统的客户端对象 * 第一个参数：HDFS的URI * 第二个参数：客户端制定的配置参数 * 第三个参数：客户端的身份，说白了就是用户名 */ fileSystem = FileSystem.get(new URI(HDFS_PATH),configuration,"hadoop"); } /* * 拷贝本地文件到HDFS文件系统 */ @Test public void copyFromLocalFile() throws Exception{ Path src = new Path("/home/hadoop/t.txt"); Path dst = new Path("/hdfsapi/test/"); fileSystem.copyFromLocalFile(src,dst); } @After public void tearDown(){ configuration = null; fileSystem = null; System.out.println("----------tearDown------"); } }
方法怎么用？还是那句 哪里不会Ctrl点哪里。
点进CopyFromLocalFile方法源码得知方法需要两个参数：本地文件的Path，和目标文件的Path，无返回值。
我们运行该测试类后进入终端使用-ls查看/hdfsapi/test目录下包含了刚刚copy进来的t.txt文件，测试成功。
[hadoop@hadoop000 ~]$ hadoop fs -ls /hdfsapi/test Found 3 items -rw-r--r-- 3 hadoop supergroup 14 2019-04-19 16:31 /hdfsapi/test/a.txt -rw-r--r-- 1 hadoop supergroup 28 2019-04-19 16:50 /hdfsapi/test/c.txt -rw-r--r-- 1 hadoop supergroup 2732 2019-04-20 19:51 /hdfsapi/test/t.txt
如果我们需要从本地拷贝一个大文件，文件越大需要等待的时间自然越长，这么漫长的等待且毫无显示严重影响用户体验。
所以在上传大文件的时候可以添加上传进度条，在fileSystem下有个create方法带有进度条的功能：
/** * Create an FSDataOutputStream at the indicated Path with write-progress * reporting. * Files are overwritten by default. * @param f the file to create * @param progress to report progress *在具有写入进度的指定路径上创建fsdataoutputstream。 *默认情况下会覆盖文件。 *@参数 f 要创建的文件 *@参数 progress 报告进度 */ public FSDataOutputStream create(Path f, Progressable progress) throws IOException { return create(f, true, getConf().getInt("io.file.buffer.size", 4096), getDefaultReplication(f), getDefaultBlockSize(f), progress); }
运行测试类，能看到打印显示，虽然全是点看起来比较抽象，但是比什么都没有到怀疑死机还是要好点儿。
setUp----------- log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................----------tearDown------ Process finished with exit code 0 ........................
我们打开终端-ls查看，上传成功。
[hadoop@hadoop000 software]$ hadoop fs -ls /hdfsapi/test Found 4 items -rw-r--r-- 3 hadoop supergroup 14 2019-04-19 16:31 /hdfsapi/test/a.txt -rw-r--r-- 1 hadoop supergroup 28 2019-04-19 16:50 /hdfsapi/test/c.txt -rw-r--r-- 1 hadoop supergroup 181367942 2019-04-20 20:10 /hdfsapi/test/jdk.zip -rw-r--r-- 1 hadoop supergroup 2732 2019-04-20 19:51 /hdfsapi/test/t.txt
能上传那就自然会问：怎么下载？直接上代码，和上面类似就不多介绍了。
/** * 拷贝HDFS文件到本地：下载 * @throws Exception */ @Test public void copyToLocalFile() throws Exception{ Path src = new Path("/hdfsapi/test/t.txt"); Path dst = new Path("/home/hadoop/app"); fileSystem.copyToLocalFile(src,dst); }
posted on
2019-04-20 20:33 Liuyt_61 阅读(
...) 评论(
...) 编辑 收藏