HDFS下载数据之源码分析-FileSystem.get(conf)_block02
接block01
来自分割线4, 调用NameNodeProxies.createNNProxyWithClientProtocol(InetSocketAddress address, Configuration conf, UserGroupInformation ugi, boolean withRetries)方法
private static ClientProtocol createNNProxyWithClientProtocol(InetSocketAddress address,
Configuration conf, UserGroupInformation ugi, boolean withRetries) throws IOException {
// 设置一个RPC protocol, 使用非默认的RpcEngine
RPC.setProtocolEngine(conf, ClientNamenodeProtocolPB.class, ProtobufRpcEngine.class);
// 获取配置文件中默认的RetryPolicy
final RetryPolicy defaultPolicy =
RetryUtils.getDefaultRetryPolicy(
conf,
DFSConfigKeys.DFS_CLIENT_RETRY_POLICY_ENABLED_KEY,
DFSConfigKeys.DFS_CLIENT_RETRY_POLICY_ENABLED_DEFAULT,
DFSConfigKeys.DFS_CLIENT_RETRY_POLICY_SPEC_KEY,
DFSConfigKeys.DFS_CLIENT_RETRY_POLICY_SPEC_DEFAULT,
SafeModeException.class);
// 获取ClientNamenodeProtocolPB这个RPC协议接口的versionID
final long version = RPC.getProtocolVersion(ClientNamenodeProtocolPB.class);
// 第一次获取ClientNamenodeProtocolPB的代理对象
ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(
ClientNamenodeProtocolPB.class, version, address, ugi, conf,
NetUtils.getDefaultSocketFactory(conf),
org.apache.hadoop.ipc.Client.getTimeout(conf), defaultPolicy)
.getProxy();
if (withRetries) { // create the proxy with retries
...
Map<String, RetryPolicy> methodNameToPolicyMap
= new HashMap<String, RetryPolicy>();
methodNameToPolicyMap.put("create", methodPolicy);
// 再次创建ClientNamenodeProtocolPB的代理对象
proxy = (ClientNamenodeProtocolPB) RetryProxy.create(
ClientNamenodeProtocolPB.class,
new DefaultFailoverProxyProvider<ClientNamenodeProtocolPB>(
ClientNamenodeProtocolPB.class, proxy),
methodNameToPolicyMap,
defaultPolicy);
}
// 返回一个proxy的包装对象
return new ClientNamenodeProtocolTranslatorPB(proxy);
}
至此, 你还记得createNNProxyWithClientProtocol()方法返回到哪里了吗?
答案是分割线4: NameNodeProxies.createNonHAProxy(Configuration conf, InetSocketAddress nnAddr, Class<T> xface, UserGroupInformation ugi, boolean withRetries)方法!
public static <T> ProxyAndInfo<T> createNonHAProxy(Configuration conf, InetSocketAddress nnAddr,
Class<T> xface, UserGroupInformation ugi, boolean withRetries) throws IOException {
...
// 创建一个namenode代理对象
proxy = (T) createNNProxyWithClientProtocol(nnAddr, conf, ugi, withRetries);
/* 分割线4, 期待着createNNProxyWithClientProtocol()方法返回 */
...
// 把proxy, dtService封装成一个ProxyAndInfo对象, 并返回
return new ProxyAndInfo<T>(proxy, dtService);
}
至此, 你还记得createNonHAProxy()方法返回到哪里了吗?
答案是分割线3: DFSClient类的构造函数里!
DFSClient(URI nameNodeUri, ClientProtocol rpcNamenode, Configuration conf, FileSystem$Statistics statistics)
public DFSClient(URI nameNodeUri, ClientProtocol rpcNamenode, Configuration conf,
FileSystem.Statistics stats) throws IOException {
...
proxyInfo = NameNodeProxies.createProxy(conf, nameNodeUri, ClientProtocol.class);
/* 分割线3, 期待着createProxy()方法返回 */
this.dtService = proxyInfo.getDelegationTokenService(); // DFSClient的成员变量dtService引用从proxyInfo中取出的dtService
this.namenode = proxyInfo.getProxy(); // DFSClient的成员变量namenode引用从proxyInfo中取出的namenode
...
}
至此! DFSClient实例彻底创建完了! 你还记得这个DFSClient实例返回到哪里了吗?
答案是分割线2: DistributedFileSystem.initialize(URI uri, Configuration conf)方法
public void initialize(URI uri, Configuration conf) throws IOException {
...
// new一个DFSClient实例,成员变量dfs引用这个DFSClient实例
this.dfs = new DFSClient(uri, conf, statistics );
/* 分割线2, 期待着new DFSClient()返回 */
...
}
至此! DistributedFileSystem实例彻底创建完了! 你还记得这个DistributedFileSystem实例返回到哪里了吗?
答案是分割线1:
FileSystem$Cache.getInternal(URI uri, Configuration conf, FileSystem$Cache$Key key)方法
private FileSystem getInternal(URI uri, Configuration conf, Key key) throws IOException{
...
fs = createFileSystem(uri, conf);
/* 分割线1, 期待着createFileSystem()方法返回 */
synchronized (this) { // refetch the lock again
/*
* 在多线程环境下, 可能另一个客户端(另一个线程)创建好了一个DistributedFileSystem实例, 并缓存到了map中
* 所以, 这时候就把当前客户端新创建的DistributedFileSystem实例注销
* 其实这是一个特殊的单例模式, 一个key映射一个DistributedFileSystem实例
*/
FileSystem oldfs = map.get(key);
if (oldfs != null) { // a file system is created while lock is releasing
fs.close(); // close the new file system
return oldfs; // return the old file system
}
/*
* now insert the new file system into the map
* 缓存当前新创建的DistributedFileSystem实例到map中
*/
fs.key = key;
map.put(key, fs);
...
return fs;
}
}
...
绕了这么一大圈, 终于返回到客户端代码中
FileSystem fs = FileSystem.get(new Configuration()); // fs = DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_542259566_1, ugi=root (auth:SIMPLE)]]
