hadoop 4.1.0 cdh4读文件源码分析
上篇文章分析了hadoop写文件的流程，既然明白了文件是怎么写入的，再来理解读就顺畅一些了。
同样的，本文主要探讨客户端的实现，同样的，我依然推荐读一下http://www.cnblogs.com/duguguiyu/archive/2009/02/22/1396034.html，读文件的大致流程如下：
不论是文件读取，还是文件的写入，主控服务器扮演的都是中介的角色。客户端把自己的需求提交给主控服务器，主控服务器挑选合适的数据服务器，介绍给客户端，让客户端和数据服务器单聊，要读要写随你们便。这种策略类似于DMA，降低了主控服务器的负载，提高了效率。。。
因此，在文件读写操作中，最主要的通信，发生在客户端与数据服务器之间。它们之间跑的协议是ClientDatanodeProtocol。从这个协议中间，你无法看到和读写相关的接口，因为，在Hadoop中，读写操作是不走RPC机制的，而是另立门户，独立搭了一套通信框架。在数据服务器一端，DataNode类中有一个DataXceiverServer类的实例，它在一个单独的线程等待请求，一旦接到，就启动一个DataXceiver的线程，处理此次请求。一个请求一个线程，对于数据服务器来说，逻辑上很简单。当下，DataXceiver支持的请求类型有六种，具体的请求包和回复包格式，请参见这里，这里，这里。在Hadoop的实现中，并没有用类来封装这些请求，而是按流的次序写下来，这给代码阅读带来挺多的麻烦，也对代码的维护带来一定的困难，不知道是出于何种考虑。。。
相比于写，文件的读取实在是一个简单的过程。在客户端DFSClient中，有一个DFSClient.DFSInputStream类。当需要读取一个文件的时候，会生成一个DFSInputStream的实例。它会先调用ClientProtocol定义getBlockLocations接口，提供给NameNode文件路径、读取位置、读取长度信息，从中取得一个LocatedBlocks类的对象，这个对象包含一组LocatedBlock，那里面有所规定位置中包含的所有数据块信息，以及数据块对应的所有数据服务器的位置信息。当读取开始后，DFSInputStream会先尝试从某个数据块对应的一组数据服务器中选出一个，进行连接。这个选取算法，在当下的实现中，非常简单，就是选出第一个未挂的数据服务器，并没有加入客户端与数据服务器相对位置的考量。读取的请求，发送到数据服务器后，自然会有DataXceiver来处理，数据被一个包一个包发送回客户端，等到整个数据块的数据都被读取完了，就会断开此链接，尝试连接下一个数据块对应的数据服务器，整个流程，依次如此反复，直到所有想读的都读取完了为止。。。
跟写文件类似，读文件的主要逻辑在DFSInputStream类中。先看下构造函数：
DFSInputStream(DFSClient dfsClient, String src, int buffersize, boolean verifyChecksum
) throws IOException, UnresolvedLinkException {
this.dfsClient = dfsClient;
this.verifyChecksum = verifyChecksum;
this.buffersize = buffersize;
this.src = src;
this.socketCache = dfsClient.socketCache;
prefetchSize = dfsClient.getConf().prefetchSize;
timeWindow = dfsClient.getConf().timeWindow;
nCachedConnRetry = dfsClient.getConf().nCachedConnRetry;
openInfo();
}
在写文件的准备工作方法openInfo中会向namenode读取被打开文件所需的所有BlockId。
LocatedBlocks newInfo = dfsClient.getLocatedBlocks(src, 0, prefetchSize);
if (DFSClient.LOG.isDebugEnabled()) {
DFSClient.LOG.debug("newInfo = " + newInfo);
}
if (newInfo == null) {
throw new IOException("Cannot open filename " + src);
}
if (locatedBlocks != null) {
Iterator<LocatedBlock> oldIter = locatedBlocks.getLocatedBlocks().iterator();
Iterator<LocatedBlock> newIter = newInfo.getLocatedBlocks().iterator();
while (oldIter.hasNext() && newIter.hasNext()) {
if (! oldIter.next().getBlock().equals(newIter.next().getBlock())) {
throw new IOException("Blocklist for " + src + " has changed!");
}
}
}
locatedBlocks = newInfo;
long lastBlockBeingWrittenLength = 0;
if (!locatedBlocks.isLastBlockComplete()) {
final LocatedBlock last = locatedBlocks.getLastLocatedBlock();
if (last != null) {
if (last.getLocations().length == 0) {
return -1;
}
final long len = readBlockLength(last);
last.getBlock().setNumBytes(len);
lastBlockBeingWrittenLength = len;
}
}
currentNode = null;
return lastBlockBeingWrittenLength;
1.dfsClient.getLocatedBlocks方法实际调用了namenode.getBlockLocations返回所有的blockId。
2.查看blockId信息是否已被cache，如没有则将cache赋值。
3.判断该文件是否isLastBlockComplete，在hadoop中写文件实际是把block写入到datanode中，而namenode是通过datanode定期的汇报得知该文件到底由哪几个block组成的。因此，在读某个文件时可能存在datanode还未汇报给namenode的情况，因此，我们在读文件时只能读到最后一个汇报的block块。
下面看下read方法。
public synchronized int read(final byte buf[], int off, int len) throws IOException {
ReaderStrategy byteArrayReader = new ByteArrayStrategy(buf);
return readWithStrategy(byteArrayReader, off, len);
}
首先会创建一个ByteArrayStrategy的reader，这种reader会将block依次读到buf数组中，hadoop还提供一个ByteBufferStrategy用来支持NIO模式的读。
然后执行readWithStrategy。
try {
// currentNode can be left as null if previous read had a checksum
// error on the same block. See HDFS-3067
if (pos > blockEnd || currentNode == null) {
currentNode = blockSeekTo(pos);
}
int realLen = (int) Math.min(len, (blockEnd - pos + 1L));
int result = readBuffer(strategy, off, realLen, corruptedBlockMap);
if (result >= 0) {
pos += result;
} else {
// got a EOS from reader though we expect more data on it.
throw new IOException("Unexpected EOS from the reader");
}
if (dfsClient.stats != null && result != -1) {
dfsClient.stats.incrementBytesRead(result);
}
return result;
1.pos指当前读文件的偏移量，首先根据pos获取当前应该读的block对象
2.根据block对象向namenode询问有哪些datanode拥有该block，选择需要读取的datanode
3.建立client-datanode的链接，创建BlockReader。
4.readBuffer方法调用不同的readStrategy的doRead方法从block中读取想要的数据。
下面对上面三步分别解释下：
1.获取block对象
public int findBlock(long offset) {
// create fake block of size 0 as a key
LocatedBlock key = new LocatedBlock(
new ExtendedBlock(), new DatanodeInfo[0], 0L, false);
key.setStartOffset(offset);
key.getBlock().setNumBytes(1);
Comparator<LocatedBlock> comp =
new Comparator<LocatedBlock>() {
// Returns 0 iff a is inside b or b is inside a
@Override
public int compare(LocatedBlock a, LocatedBlock b) {
long aBeg = a.getStartOffset();
long bBeg = b.getStartOffset();
long aEnd = aBeg + a.getBlockSize();
long bEnd = bBeg + b.getBlockSize();
if(aBeg <= bBeg && bEnd <= aEnd
|| bBeg <= aBeg && aEnd <= bEnd)
return 0; // one of the blocks is inside the other
if(aBeg < bBeg)
return -1; // a's left bound is to the left of the b's
return 1;
}
};
return Collections.binarySearch(blocks, key, comp);
}
获得Block对象的核心方法是findBlock方法。通过比较各个block在整个文件中的位移来确定当前位移在哪个block中
2.获得datanode
static DatanodeInfo bestNode(DatanodeInfo nodes[],
AbstractMap<DatanodeInfo, DatanodeInfo> deadNodes)
throws IOException {
if (nodes != null) {
for (int i = 0; i < nodes.length; i++) {
if (!deadNodes.containsKey(nodes[i])) {
return nodes[i];
}
}
}
throw new IOException("No live nodes contain current block");
}
获得best的datanode很简单。就是遍历该block所有的datanode，按顺序取最前面的。不过，其实在namenode端返回datanodeList时就是按照优先级顺序返回的。
private DNAddrPair chooseDataNode(LocatedBlock block)
throws IOException {
while (true) {
DatanodeInfo[] nodes = block.getLocations();
try {
DatanodeInfo chosenNode = bestNode(nodes, deadNodes);
final String dnAddr =
chosenNode.getXferAddr(dfsClient.connectToDnViaHostname());
if (DFSClient.LOG.isDebugEnabled()) {
DFSClient.LOG.debug("Connecting to datanode " + dnAddr);
}
InetSocketAddress targetAddr = NetUtils.createSocketAddr(dnAddr);
return new DNAddrPair(chosenNode, targetAddr);
} catch (IOException ie) {
String blockInfo = block.getBlock() + " file=" + src;
if (failures >= dfsClient.getMaxBlockAcquireFailures()) {
throw new BlockMissingException(src, "Could not obtain block: " + blockInfo,
block.getStartOffset());
}
if (nodes == null || nodes.length == 0) {
DFSClient.LOG.info("No node available for block: " + blockInfo);
}
DFSClient.LOG.info("Could not obtain block " + block.getBlock()
+ " from any node: " + ie
+ ". Will get new block locations from namenode and retry...");
try {
// Introducing a random factor to the wait time before another retry.
// The wait time is dependent on # of failures and a random factor.
// At the first time of getting a BlockMissingException, the wait time
// is a random number between 0..3000 ms. If the first retry
// still fails, we will wait 3000 ms grace period before the 2nd retry.
// Also at the second retry, the waiting window is expanded to 6000 ms
// alleviating the request rate from the server. Similarly the 3rd retry
// will wait 6000ms grace period before retry and the waiting window is
// expanded to 9000ms.
double waitTime = timeWindow * failures + // grace period for the last round of attempt
timeWindow * (failures + 1) * DFSUtil.getRandom().nextDouble(); // expanding time window for each failure
DFSClient.LOG.warn("DFS chooseDataNode: got # " + (failures + 1) + " IOException, will wait for " + waitTime + " msec.");
Thread.sleep((long)waitTime);
} catch (InterruptedException iex) {
}
deadNodes.clear(); //2nd option is to remove only nodes[blockId]
openInfo();
block = getBlockAt(block.getStartOffset(), false);
failures++;
continue;
}
}
}
这边主要是创建datanode的socket失败的重试，采取了graceful的sleep方式，可以学习一下，不过好像实际sleep的方式跟注释中描述的不一样。
3.创建BlockReader
// Can't local read a block under construction, see HDFS-2757
if (dfsClient.shouldTryShortCircuitRead(dnAddr) &&
!blockUnderConstruction()) {
return DFSClient.getLocalBlockReader(dfsClient.conf, src, block,
blockToken, chosenNode, dfsClient.hdfsTimeout, startOffset,
dfsClient.connectToDnViaHostname());
}
ShortCircuitRead是hadoop的一个优化。在client与datanode在同一台机器时会直接读本地文件而不是通过socket向datanode读取block。
// Allow retry since there is no way of knowing whether the cached socket
// is good until we actually use it.
for (int retries = 0; retries <= nCachedConnRetry && fromCache; ++retries) {
SocketAndStreams sockAndStreams = null;
// Don't use the cache on the last attempt - it's possible that there
// are arbitrarily many unusable sockets in the cache, but we don't
// want to fail the read.
if (retries < nCachedConnRetry) {
sockAndStreams = socketCache.get(dnAddr);
}
Socket sock;
if (sockAndStreams == null) {
fromCache = false;
sock = dfsClient.socketFactory.createSocket();
// TCP_NODELAY is crucial here because of bad interactions between
// Nagle's Algorithm and Delayed ACKs. With connection keepalive
// between the client and DN, the conversation looks like:
// 1. Client -> DN: Read block X
// 2. DN -> Client: data for block X
// 3. Client -> DN: Status OK (successful read)
// 4. Client -> DN: Read block Y
// The fact that step #3 and #4 are both in the client->DN direction
// triggers Nagling. If the DN is using delayed ACKs, this results
// in a delay of 40ms or more.
//
// TCP_NODELAY disables nagling and thus avoids this performance
// disaster.
sock.setTcpNoDelay(true);
NetUtils.connect(sock, dnAddr,
dfsClient.getRandomLocalInterfaceAddr(),
dfsClient.getConf().socketTimeout);
sock.setSoTimeout(dfsClient.getConf().socketTimeout);
} else {
sock = sockAndStreams.sock;
}
try {
// The OP_READ_BLOCK request is sent as we make the BlockReader
BlockReader reader =
BlockReaderFactory.newBlockReader(dfsClient.getConf(),
sock, file, block,
blockToken,
startOffset, len,
bufferSize, verifyChecksum,
clientName,
dfsClient.getDataEncryptionKey(),
sockAndStreams == null ? null : sockAndStreams.ioStreams);
return reader;
} catch (IOException ex) {
// Our socket is no good.
DFSClient.LOG.debug("Error making BlockReader. Closing stale " + sock, ex);
if (sockAndStreams != null) {
sockAndStreams.close();
} else {
sock.close();
}
err = ex;
}
首先对socket进行了初始化。这边设置了TCPNODELAY。因为client-datanode的交互是严格时序的。如果不设置client会非常慢。
BlockReaderFactory.newBlockReader创建BlockReader对象。后面就通过这个reader来依次读出block的内容。
4.BlockReader读block
public synchronized int read(byte[] buf, int off, int len)
throws IOException {
if (curDataSlice == null || curDataSlice.remaining() == 0 && bytesNeededToFinish > 0) {
readNextPacket();
}
if (curDataSlice.remaining() == 0) {
// we're at EOF now
return -1;
}
int nRead = Math.min(curDataSlice.remaining(), len);
curDataSlice.get(buf, off, nRead);
return nRead;
//Read packet headers.
packetReceiver.receiveNextPacket(in);
PacketHeader curHeader = packetReceiver.getHeader();
curDataSlice = packetReceiver.getDataSlice();
assert curDataSlice.capacity() == curHeader.getDataLen();
if (LOG.isTraceEnabled()) {
LOG.trace("DFSClient readNextPacket got header " + curHeader);
}
// Sanity check the lengths
if (!curHeader.sanityCheck(lastSeqNo)) {
throw new IOException("BlockReader: error in packet header " +
curHeader);
}
if (curHeader.getDataLen() > 0) {
int chunks = 1 + (curHeader.getDataLen() - 1) / bytesPerChecksum;
int checksumsLen = chunks * checksumSize;
assert packetReceiver.getChecksumSlice().capacity() == checksumsLen :
"checksum slice capacity=" + packetReceiver.getChecksumSlice().capacity() +
" checksumsLen=" + checksumsLen;
lastSeqNo = curHeader.getSeqno();
if (verifyChecksum && curDataSlice.remaining() > 0) {
// N.B.: the checksum error offset reported here is actually
// relative to the start of the block, not the start of the file.
// This is slightly misleading, but preserves the behavior from
// the older BlockReader.
checksum.verifyChunkedSums(curDataSlice,
packetReceiver.getChecksumSlice(),
filename, curHeader.getOffsetInBlock());
}
bytesNeededToFinish -= curHeader.getDataLen();
}
// First packet will include some data prior to the first byte
// the user requested. Skip it.
if (curHeader.getOffsetInBlock() < startOffset) {
int newPos = (int) (startOffset - curHeader.getOffsetInBlock());
curDataSlice.position(newPos);
}
// If we've now satisfied the whole client read, read one last packet
// header, which should be empty
if (bytesNeededToFinish <= 0) {
readTrailingEmptyPacket();
if (verifyChecksum) {
sendReadResult(Status.CHECKSUM_OK);
} else {
sendReadResult(Status.SUCCESS);
}
}
这边调用了RemoteBlockReader2的read方法读取block。这边的逻辑也不简单，datanode把block按一个个packet发送过来，每发送一个packet都需要checkSum校验数据正确性和检查packet的头数据，把实际data数据放到curDataSlice中，数据正确后会发送response给datanode，只有收到了client的ack信息datanode才会发送下一个packet。
刚读这边代码有一个奇怪的地方。如果用户从block中读一段数据到字符数据中时，如果读的长度超过block的大小，超过的部分不会被读到。不过后来仔细看了下，DFSInputStream的readWithStrategy方法每次都只会读一个int型，所以就不会出现此问题了。
总结一下：
1.客户端采用了read，ack的强时序模式而且没有用线程来receive数据，保证了读的正确性，但也稍微降低了读的效率。
2.socket采用了NIO提高了效率，这可能是与较早版本最大的改进。
3.虽然说0.92版本shortCircuit没有实现，但看cdh4代码中是有的。不知道如何测试是否能够使用。