《Hadoop权威指南》学习笔记（一）
本博文是我学习《Hadoop权威指南》3.5节的笔记，主要是里面范例程序的实现，部分实现有修改
1 从Hadoop读取数据
首先新建一个文本文件test.txt作为测试
hadoop fs -mkdir /poems //在Hadoop集群上新建一个目录poems
hadoop fs -copyFromLocal test.txt /poems/test.txt //将本地test.txt文件上传到Hadoop集群
hadoop fs -ls /poems //检查是否成功上传
上传无误就可以在poems目录下看见刚才上传的文件，准备工作完成，接下来用两种方法读取Hadoop集群上的test.txt文件
1.1 URL读取
新建类URLCat如下
package com.tuan.hadoopLearn.hdfs;
import java.net.URL;
import java.io.InputStream;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
public class URLCat {
static {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
}
public static void main(String[] args) {
InputStream in = null;
try {
in = new URL(args[0]).openStream();
IOUtils.copyBytes(in, System.out, 4096, false);
} catch (Exception e) {
System.out.println(e);
} finally {
IOUtils.closeStream(in);
}
}
}maven install后，进入工程jar包目录下（一般是.m2目录下），cmd执行命令，注意两个路径要写对，一个是URLCat类的路径，一个是Hadoop集群的路径
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.URLCat hdfs:/poems/test.txt //执行jar包下的URLCat类这里可能会出现找不到或无法加载主类的错误，一般有两个原因，一个是CLASSPATH环境变量没有配置，我使用的Java1.8配置CLASSPATH应该为配置为“.;%JAVA_HOME\lib\dt.jar;%JAVA_HOME%\lib\tools.jar”，最前面有一个“.”，表示当前目录；另一个原因是类的路径没有写完整
执行成功后就可以通过URLCat类读取到刚才上传的test.txt文件并在控制台打印
1.2 FileSystem读取
类似地，写一个FileSystemCat类
package com.tuan.hadoopLearn.hdfs;
import java.io.InputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class FileSystemCat {
public static void main(String[] args) {
String uri = args[0];
Configuration conf = new Configuration();
InputStream in = null;
try {
FileSystem fs = FileSystem.get(URI.create(uri), conf);
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false);
} catch (Exception e) {
System.out.println(e);
} finally {
IOUtils.closeStream(in);
}
}
}同样maven install后，cmd输入下列命令运行，可以看到输出test.txt内容
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.FileSystemCat hdfs:/poems/test.txt
1.3 读取位置定位
将java.io的InputStream替换为hadoop的FsDataInputStream，可以实现按指定偏移量读取文件的功能，新建FileSystemDoubleCat类如下
package com.tuan.hadoopLearn.hdfs;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class FileSystemDoubleCat {
public static void main(String[] args) {
String uri = args[0];
Configuration conf = new Configuration();
FSDataInputStream in = null;
try {
FileSystem fs = FileSystem.get(URI.create(uri), conf);
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false);
System.out.println("\n------------------------------------------------------------------");
in.seek(4);
IOUtils.copyBytes(in, System.out, 4096, false);
} catch (Exception e) {
System.out.println(e);
} finally {
IOUtils.closeStream(in);
}
}
}同样执行类似的cmd命令，结果如图，第二次输出的时候开始的4个字母没有输出，因为执行了seek(4)
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.FileSystemDoubleCat hdfs:/poems/test.txt
2 向Hadoop写入数据
在本地新建一个poem1.txt文件，准备等会通过Java传到Hadoop集群上
新建类FileCopyWithProgress
package com.tuan.hadoopLearn.hdfs;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;
public class FileCopyWithProgress {
public static void main(String[] args) {
String localSrc = args[0];
String dst = args[1];
InputStream in = null;
OutputStream out = null;
Configuration conf = new Configuration();
try {
in = new BufferedInputStream(new FileInputStream(localSrc));
FileSystem fs = FileSystem.get(URI.create(dst), conf);
out = fs.create(new Path(dst), new Progressable() {
@Override
public void progress() { //回调函数，显示上传过程
System.out.print(".");
}
});
IOUtils.copyBytes(in, out, 4096, true);
} catch (Exception e) {
System.out.println(e);
}
}
}输入cmd命令，将文件poems.txt传到hadoop集群上
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.FileCopyWithProgress C:\Users\Lenovo\Desktop\poem1.txt hdfs:/poems/poem1.txt
hadoop fs -ls /poems //检查是否文件成功上传
3 文件信息查询
3.1 读取文件状态
这个和书上的例子不太一样，书上做了一个测试小集群，这里直接在集群上测试，新建类ShowFileStatusTest
package com.tuan.hadoopLearn.hdfs;
import java.io.IOException;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileStatus;
public class ShowFileStatusTest {
public static void main(String[] args) {
String uri = args[0];
try {
FileSystem fs = FileSystem.get(URI.create(uri), new Configuration());
FileStatus stat = fs.getFileStatus(new Path(uri));
if (stat.isDir()) {
System.out.println(uri + " is a directory");
} else {
System.out.println(uri + " is a file");
}
System.out.println("The path is " + stat.getPath());
System.out.println("The length is " + stat.getLen());
System.out.println("The modification time is " + stat.getModificationTime());
System.out.println("The permission is " + stat.getPermission().toString());
} catch (IOException e) {
e.printStackTrace();
}
}
}
输入cmd命令
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.ShowFileStatusTest hdfs:/poems //读取poems目录信息
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.ShowFileStatusTest hdfs:/poems/poem1.txt //读取poem1.txt文件信息
3.2 列出目录下文件
这次同时列出两个指定目录下的文件，再新建一个jokes目录，放入一个joke1.txt
写好类ListStatus
package com.tuan.hadoopLearn.hdfs;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
public class ListStatus {
public static void main(String[] args) {
try {
FileSystem fs = FileSystem.get( new Configuration());
Path[] paths = new Path[args.length];
for (int i = 0; i < args.length; i ++) {
paths[i] = new Path(args[i]);
}
FileStatus[] stats = fs.listStatus(paths);
Path[] listPaths = FileUtil.stat2Paths(stats);
for (Path p : listPaths) {
System.out.println(p);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
cmd命令
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.ListStatus hdfs:/poems hdfs:/jokes
3.3 目录过滤
新建类RegexExcludedPathFilter，将指定目录下符合条件的目录过滤并将其他目录列出
package com.tuan.hadoopLearn.hdfs;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathFilter;
public class RegexExcludedPathFilter implements PathFilter{
private final String reg;
public RegexExcludedPathFilter(String reg) {
this.reg = reg;
}
@Override
public boolean accept(Path path) {
return !path.toString().matches(reg);
}
public static void main(String[] args) {
String uri = args[0];
String reg = args[1];
try {
FileSystem fs = FileSystem.get(new Configuration());
FileStatus[] stats = fs.globStatus(new Path(uri), new RegexExcludedPathFilter(reg));
Path[] listPaths = FileUtil.stat2Paths(stats);
for (Path p : listPaths) {
System.out.println(p);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
输入cmd命令，这里我卡了很久，本来poems目录下有两个文件poem1.txt和test.txt，我试图用/poems/t.*过滤掉test.txt，一直不成功，后来改成.*/poems/t.*就成功了，似乎globStatus()两个参数目录的输入方式不同，PathFilter的目录是完成的目录，这仅是本人不负责任的推测，留待以后继续研究
hadoop jar hadoopLearn-0.0.1-SNAPSHOT.jar com.tuan.hadoopLearn.RegexExcludedPathFilter /poems/* .*/poems/t.*