阅读更多
redis 学习问题总结 |
http://aperise.iteye.com/blog/2310639 |
ehcache memcached redis 缓存技术总结 |
http://aperise.iteye.com/blog/2296219 |
redis-stat 离线安装 |
http://aperise.iteye.com/blog/2310254 |
redis cluster 非ruby方式启动 |
http://aperise.iteye.com/blog/2310254 |
redis-sentinel安装部署 |
http://aperise.iteye.com/blog/2342693 |
spring-data-redis使用 |
http://aperise.iteye.com/blog/2342615 |
redis客户端redisson实战 |
http://aperise.iteye.com/blog/2396196 |
redisson-2.10.4源代码分析 |
http://aperise.iteye.com/blog/2400528 |
tcmalloc jemalloc libc选择 |
http://blog.csdn.net/u010994304/article/details/49906819 |
1.RedissonClient一主两从部署时连接池组成
对如下图的主从部署(1主2从):
redisson纯java操作代码如下:
- Config config = new Config();
- config.useMasterSlaveServers()
-
-
-
-
-
-
-
-
- .setMasterAddress("redis://192.168.29.24:6379") // 设置redis主节点
- .addSlaveAddress("redis://192.168.29.24:7000") // 设置redis从节点
- .addSlaveAddress("redis://192.168.29.24:7001"); // 设置redis从节点
- RedissonClient redisson = Redisson.create(config);
上面代码执行完毕后,如果在redis服务端所在服务器执行以下linux命令:
- #6379上建立了10个连接
- netstat -ant |grep 6379|grep ESTABLISHED
- #7000上建立了11个连接
- netstat -ant |grep 7000|grep ESTABLISHED
- #7001上建立了11个连接
- netstat -ant |grep 7001|grep ESTABLISHED
你会发现redisson连接到redis服务端总计建立了32个连接,其中masterpool占据10个连接,slavepool占据20个连接,另外pubSubConnectionPool占据2个连接,连接池中池化对象分布如下图:
- MasterConnectionPool:默认针对每个不同的IP+port组合,初始化10个对象,最大可扩展至64个,因为只有一个master,所以上图创建了10个连接;
- MasterPubSubConnectionPool:默认针对每个不同的IP+port组合,初始化1个对象,最大可扩展至50个,因为默认SubscriptionMode=SubscriptionMode.SLAVE,所以master上不会创建连接池,所以上图MasterPubSubConnectionPool里没有创建任何连接;
- SlaveConnectionPool:默认针对每个不同的IP+port组合,初始化10个对象,最大可扩展至64个,因为有两个slave,每个slave上图创建了10个连接,总计创建了20个连接;
- PubSubConnectionPool:默认针对每个不同的IP+port组合,初始化1个对象,最大可扩展至50个,因为有两个slave,每个slave上图创建了1个连接,总计创建了2个连接。
从上图可以看出,连接池是针对每个IP端口都有一个独立的池,连接池也按照主从进行划分,那么这些连接池是在哪里初始化的?如何初始化的?读操作和写操作如何进行的?这就是今天要解答的问题,要解答这些问题最好还是查看redisson的源码。
2.Redisson初始化连接池源码分析
2.1 Redisson.java
RedissonClient.java是一个接口类,它的实现类是Redisson.java,对于Redisson.java的介绍先以一张Redisson的4大组件关系图开始,如下图:
对Redisson.java的代码注释如下:
-
-
-
-
-
- public static RedissonClient create(Config config) {
-
- Redisson redisson = new Redisson(config);
- if (config.isRedissonReferenceEnabled()) {
- redisson.enableRedissonReferenceSupport();
- }
- return redisson;
- }
-
-
-
-
-
-
- protected Redisson(Config config) {
-
- this.config = config;
-
- Config configCopy = new Config(config);
-
-
- connectionManager = ConfigSupport.createConnectionManager(configCopy);
-
- evictionScheduler = new EvictionScheduler(connectionManager.getCommandExecutor());
-
- codecProvider = configCopy.getCodecProvider();
-
- resolverProvider = configCopy.getResolverProvider();
- }
其中与连接池相关的就是ConnectionManager,ConnectionManager的初始化转交工具类ConfigSupport.java进行,ConfigSupport.java会根据部署方式(主从模式、单机模式、哨兵模式、集群模式、亚马逊云模式、微软云模式)的不同而分别进行。
2.2 ConfigSupport.java
这里现将ConfigSupport.java创建ConnectionManager的核心代码注释如下:
-
-
-
-
-
- public static ConnectionManager createConnectionManager(Config configCopy) {
- if (configCopy.getMasterSlaveServersConfig() != null) {
- validate(configCopy.getMasterSlaveServersConfig());
- return new MasterSlaveConnectionManager(configCopy.getMasterSlaveServersConfig(), configCopy);
- } else if (configCopy.getSingleServerConfig() != null) {
- validate(configCopy.getSingleServerConfig());
- return new SingleConnectionManager(configCopy.getSingleServerConfig(), configCopy);
- } else if (configCopy.getSentinelServersConfig() != null) {
- validate(configCopy.getSentinelServersConfig());
- return new SentinelConnectionManager(configCopy.getSentinelServersConfig(), configCopy);
- } else if (configCopy.getClusterServersConfig() != null) {
- validate(configCopy.getClusterServersConfig());
- return new ClusterConnectionManager(configCopy.getClusterServersConfig(), configCopy);
- } else if (configCopy.getElasticacheServersConfig() != null) {
- validate(configCopy.getElasticacheServersConfig());
- return new ElasticacheConnectionManager(configCopy.getElasticacheServersConfig(), configCopy);
- } else if (configCopy.getReplicatedServersConfig() != null) {
- validate(configCopy.getReplicatedServersConfig());
- return new ReplicatedConnectionManager(configCopy.getReplicatedServersConfig(), configCopy);
- } else if (configCopy.getConnectionManager() != null) {
- return configCopy.getConnectionManager();
- }else {
- throw new IllegalArgumentException("server(s) address(es) not defined!");
- }
- }
上面可以看到根据传入的配置Config.java的不同,会分别创建不同的ConnectionManager的实现类。
2.3 MasterSlaveConnectionManager.java
这里开始介绍ConnectionManager,ConnectionManager.java是一个接口类,它有6个实现类,分别对应着不同的部署模式(主从模式、单机模式、哨兵模式、集群模式、亚马逊云模式、微软云模式),如下如所示:
这里以主从部署方式进行讲解,先通过一张图了解MasterSlaveConnectionManager的组成:
上图中最终要的组件要数MasterSlaveEntry,在后面即将进行介绍,这里注释MasterSlaveConnectionManager.java的核心代码如下:
-
-
-
-
-
- public MasterSlaveConnectionManager(MasterSlaveServersConfig cfg, Config config) {
-
- this(config);
-
- initTimer(cfg);
- this.config = cfg;
-
- initSingleEntry();
- }
-
-
-
-
- public MasterSlaveConnectionManager(Config cfg) {
-
- Version.logVersion();
-
- if (cfg.isUseLinuxNativeEpoll()) {
- if (cfg.getEventLoopGroup() == null) {
-
- this.group = new EpollEventLoopGroup(cfg.getNettyThreads(), new DefaultThreadFactory("redisson-netty"));
- } else {
- this.group = cfg.getEventLoopGroup();
- }
- this.socketChannelClass = EpollSocketChannel.class;
- } else {
- if (cfg.getEventLoopGroup() == null) {
-
- this.group = new NioEventLoopGroup(cfg.getNettyThreads(), new DefaultThreadFactory("redisson-netty"));
- } else {
- this.group = cfg.getEventLoopGroup();
- }
- this.socketChannelClass = NioSocketChannel.class;
- }
- if (cfg.getExecutor() == null) {
-
-
-
- int threads = Runtime.getRuntime().availableProcessors() * 2;
- if (cfg.getThreads() != 0) {
- threads = cfg.getThreads();
- }
- executor = Executors.newFixedThreadPool(threads, new DefaultThreadFactory("redisson"));
- } else {
- executor = cfg.getExecutor();
- }
-
- this.cfg = cfg;
- this.codec = cfg.getCodec();
-
- this.shutdownPromise = newPromise();
-
- this.commandExecutor = new CommandSyncService(this);
- }
-
-
-
-
- protected void initTimer(MasterSlaveServersConfig config) {
-
- int[] timeouts = new int[]{config.getRetryInterval(), config.getTimeout(), config.getReconnectionTimeout()};
- Arrays.sort(timeouts);
- int minTimeout = timeouts[0];
-
- if (minTimeout % 100 != 0) {
- minTimeout = (minTimeout % 100) / 2;
- } else if (minTimeout == 100) {
- minTimeout = 50;
- } else {
- minTimeout = 100;
- }
-
- timer = new HashedWheelTimer(Executors.defaultThreadFactory(), minTimeout, TimeUnit.MILLISECONDS, 1024);
-
-
- try {
- Field leakField = HashedWheelTimer.class.getDeclaredField("leak");
- leakField.setAccessible(true);
- leakField.set(timer, null);
- } catch (Exception e) {
- throw new IllegalStateException(e);
- }
-
- connectionWatcher = new IdleConnectionWatcher(this, config);
- }
-
-
-
-
- protected void initSingleEntry() {
- try {
-
- HashSet slots = new HashSet();
- slots.add(singleSlotRange);
-
- MasterSlaveEntry entry;
- if (config.checkSkipSlavesInit()) {
- entry = new SingleEntry(slots, this, config);
- RFuture f = entry.setupMasterEntry(config.getMasterAddress());
- f.syncUninterruptibly();
- } else {
- entry = createMasterSlaveEntry(config, slots);
- }
-
- for (int slot = singleSlotRange.getStartSlot(); slot < singleSlotRange.getEndSlot() + 1; slot++) {
- addEntry(slot, entry);
- }
-
- if (config.getDnsMonitoringInterval() != -1) {
- dnsMonitor = new DNSMonitor(this, Collections.singleton(config.getMasterAddress()),
- config.getSlaveAddresses(), config.getDnsMonitoringInterval());
- dnsMonitor.start();
- }
- } catch (RuntimeException e) {
- stopThreads();
- throw e;
- }
- }
-
-
-
-
-
-
- protected MasterSlaveEntry createMasterSlaveEntry(MasterSlaveServersConfig config, HashSet slots) {
-
- MasterSlaveEntry entry = new MasterSlaveEntry(slots, this, config);
-
- List> fs = entry.initSlaveBalancer(java.util.Collections.emptySet());
- for (RFuture future : fs) {
- future.syncUninterruptibly();
- }
- 主节点连接池MasterConnectionPool和MasterPubSubConnectionPool的默认的最小连接数初始化
- RFuture f = entry.setupMasterEntry(config.getMasterAddress());
- f.syncUninterruptibly();
- return entry;
- }
上面个人觉得有两处代码值得我们特别关注,特别说明如下:
- entry.initSlaveBalancer:从节点连接池SlaveConnectionPool和PubSubConnectionPool的默认的最小连接数初始化。
- entry.setupMasterEntry:主节点连接池MasterConnectionPool和MasterPubSubConnectionPool的默认的最小连接数初始化。
2.4 MasterSlaveEntry.java
用一张图来解释MasterSlaveEntry的组件如下:
MasterSlaveEntry.java里正是我们一直在寻找着的四个连接池MasterConnectionPool、MasterPubSubConnectionPool、SlaveConnectionPool和PubSubConnectionPool,这里注释MasterSlaveEntry.java的核心代码如下:
-
-
-
-
-
-
- public MasterSlaveEntry(Set slotRanges, ConnectionManager connectionManager, MasterSlaveServersConfig config) {
-
- for (ClusterSlotRange clusterSlotRange : slotRanges) {
- for (int i = clusterSlotRange.getStartSlot(); i < clusterSlotRange.getEndSlot() + 1; i++) {
- slots.add(i);
- }
- }
-
- this.connectionManager = connectionManager;
-
- this.config = config;
-
-
-
-
- slaveBalancer = new LoadBalancerManager(config, connectionManager, this);
-
- writeConnectionHolder = new MasterConnectionPool(config, connectionManager, this);
-
- pubSubConnectionHolder = new MasterPubSubConnectionPool(config, connectionManager, this);
- }
-
-
-
-
-
-
- public List> initSlaveBalancer(Collection disconnectedNodes) {
-
- boolean freezeMasterAsSlave = !config.getSlaveAddresses().isEmpty() && !config.checkSkipSlavesInit() && disconnectedNodes.size() < config.getSlaveAddresses().size();
-
- List> result = new LinkedList>();
-
- RFuture f = addSlave(config.getMasterAddress(), freezeMasterAsSlave, NodeType.MASTER);
- result.add(f);
-
-
-
- for (URI address : config.getSlaveAddresses()) {
- f = addSlave(address, disconnectedNodes.contains(address), NodeType.SLAVE);
- result.add(f);
- }
- return result;
- }
-
-
-
-
-
-
-
-
- private RFuture addSlave(URI address, boolean freezed, NodeType nodeType) {
-
- RedisClient client = connectionManager.createClient(NodeType.SLAVE, address);
- ClientConnectionsEntry entry = new ClientConnectionsEntry(client,
- this.config.getSlaveConnectionMinimumIdleSize(),
- this.config.getSlaveConnectionPoolSize(),
- this.config.getSubscriptionConnectionMinimumIdleSize(),
- this.config.getSubscriptionConnectionPoolSize(), connectionManager, nodeType);
-
- if (freezed) {
- synchronized (entry) {
- entry.setFreezed(freezed);
- entry.setFreezeReason(FreezeReason.SYSTEM);
- }
- }
-
- return slaveBalancer.add(entry);
- }
-
-
-
-
-
-
- public RFuture setupMasterEntry(URI address) {
-
- RedisClient client = connectionManager.createClient(NodeType.MASTER, address);
- masterEntry = new ClientConnectionsEntry(
- client,
- config.getMasterConnectionMinimumIdleSize(),
- config.getMasterConnectionPoolSize(),
- config.getSubscriptionConnectionMinimumIdleSize(),
- config.getSubscriptionConnectionPoolSize(),
- connectionManager,
- NodeType.MASTER);
-
-
- if (config.getSubscriptionMode() == SubscriptionMode.MASTER) {
-
- RFuture f = writeConnectionHolder.add(masterEntry);
- RFuture s = pubSubConnectionHolder.add(masterEntry);
- return CountListener.create(s, f);
- }
-
-
- return writeConnectionHolder.add(masterEntry);
- }
上面代码个人觉得有四个地方值得我们特别关注,它们是一个连接池创建对象的入口,列表如下:
- writeConnectionHolder.add(masterEntry):其实writeConnectionHolder的类型就是MasterConnectionPool,这里是连接池MasterConnectionPool里添加对象
- pubSubConnectionHolder.add(masterEntry):其实pubSubConnectionHolder的类型是MasterPubSubConnectionPool,这里是连接池MasterPubSubConnectionPool添加对象
- slaveConnectionPool.add(entry):这里是连接池SlaveConnectionPool里添加对象
- pubSubConnectionPool.add(entry):这里是连接池PubSubConnectionPool里添加对象
2.5 LoadBalancerManager.java
图解LoadBalancerManager.java的内部组成如下:
LoadBalancerManager.java里面有着从节点相关的两个重要的连接池SlaveConnectionPool和PubSubConnectionPool,这里注释LoadBalancerManager.java的核心代码如下:
-
-
-
-
-
-
- public LoadBalancerManager(MasterSlaveServersConfig config, ConnectionManager connectionManager, MasterSlaveEntry entry) {
-
- this.connectionManager = connectionManager;
-
- slaveConnectionPool = new SlaveConnectionPool(config, connectionManager, entry);
-
- pubSubConnectionPool = new PubSubConnectionPool(config, connectionManager, entry);
- }
-
-
-
-
-
- public RFuture add(final ClientConnectionsEntry entry) {
- final RPromise result = connectionManager.newPromise();
-
- FutureListener listener = new FutureListener() {
- AtomicInteger counter = new AtomicInteger(2);
- @Override
- public void operationComplete(Future future) throws Exception {
- if (!future.isSuccess()) {
- result.tryFailure(future.cause());
- return;
- }
- if (counter.decrementAndGet() == 0) {
- String addr = entry.getClient().getIpAddr();
- ip2Entry.put(addr, entry);
- result.trySuccess(null);
- }
- }
- };
-
- RFuture slaveFuture = slaveConnectionPool.add(entry);
- slaveFuture.addListener(listener);
-
- RFuture pubSubFuture = pubSubConnectionPool.add(entry);
- pubSubFuture.addListener(listener);
- return result;
- }
至此,我们已经了解了开篇提到的四个连接池是在哪里创建的。
3. Redisson的4类连接池
这里我们来详细介绍下Redisson的连接池实现类,Redisson里有4种连接池,它们是MasterConnectionPool、MasterPubSubConnectionPool、SlaveConnectionPool和PubSubConnectionPool,它们的父类都是ConnectionPool,其类继承关系图如下:
通过上图我们了解了ConnectionPool类的继承关系图,再来一张图来了解下ConnectionPool.java类的组成,如下:
好了,再来图就有点啰嗦了,注释ConnectionPool.java代码如下:
- abstract class ConnectionPoolextends RedisConnection> {
- private final Logger log = LoggerFactory.getLogger(getClass());
-
-
-
-
-
- protected final List entries = new CopyOnWriteArrayList();
-
- final ConnectionManager connectionManager;
-
- final MasterSlaveServersConfig config;
-
- final MasterSlaveEntry masterSlaveEntry;
-
-
- public ConnectionPool(MasterSlaveServersConfig config, ConnectionManager connectionManager, MasterSlaveEntry masterSlaveEntry) {
- this.config = config;
- this.masterSlaveEntry = masterSlaveEntry;
- this.connectionManager = connectionManager;
- }
-
-
- public RFuture add(final ClientConnectionsEntry entry) {
- final RPromise promise = connectionManager.newPromise();
- promise.addListener(new FutureListener() {
- @Override
- public void operationComplete(Future future) throws Exception {
- entries.add(entry);
- }
- });
- initConnections(entry, promise, true);
- return promise;
- }
-
-
- private void initConnections(final ClientConnectionsEntry entry, final RPromise initPromise, boolean checkFreezed) {
- final int minimumIdleSize = getMinimumIdleSize(entry);
-
- if (minimumIdleSize == 0 || (checkFreezed && entry.isFreezed())) {
- initPromise.trySuccess(null);
- return;
- }
-
- final AtomicInteger initializedConnections = new AtomicInteger(minimumIdleSize);
- int startAmount = Math.min(50, minimumIdleSize);
- final AtomicInteger requests = new AtomicInteger(startAmount);
- for (int i = 0; i < startAmount; i++) {
- createConnection(checkFreezed, requests, entry, initPromise, minimumIdleSize, initializedConnections);
- }
- }
-
-
- private void createConnection(final boolean checkFreezed, final AtomicInteger requests, final ClientConnectionsEntry entry, final RPromise initPromise,
- final int minimumIdleSize, final AtomicInteger initializedConnections) {
-
- if ((checkFreezed && entry.isFreezed()) || !tryAcquireConnection(entry)) {
- int totalInitializedConnections = minimumIdleSize - initializedConnections.get();
- Throwable cause = new RedisConnectionException(
- "Unable to init enough connections amount! Only " + totalInitializedConnections + " from " + minimumIdleSize + " were initialized. Server: "
- + entry.getClient().getAddr());
- initPromise.tryFailure(cause);
- return;
- }
-
- acquireConnection(entry, new Runnable() {
-
- @Override
- public void run() {
- RPromise promise = connectionManager.newPromise();
- createConnection(entry, promise);
- promise.addListener(new FutureListener() {
- @Override
- public void operationComplete(Future future) throws Exception {
- if (future.isSuccess()) {
- T conn = future.getNow();
-
- releaseConnection(entry, conn);
- }
-
- releaseConnection(entry);
-
- if (!future.isSuccess()) {
- int totalInitializedConnections = minimumIdleSize - initializedConnections.get();
- String errorMsg;
- if (totalInitializedConnections == 0) {
- errorMsg = "Unable to connect to Redis server: " + entry.getClient().getAddr();
- } else {
- errorMsg = "Unable to init enough connections amount! Only " + totalInitializedConnections
- + " from " + minimumIdleSize + " were initialized. Redis server: " + entry.getClient().getAddr();
- }
- Throwable cause = new RedisConnectionException(errorMsg, future.cause());
- initPromise.tryFailure(cause);
- return;
- }
-
- int value = initializedConnections.decrementAndGet();
- if (value == 0) {
- log.info("{} connections initialized for {}", minimumIdleSize, entry.getClient().getAddr());
- if (!initPromise.trySuccess(null)) {
- throw new IllegalStateException();
- }
- } else if (value > 0 && !initPromise.isDone()) {
- if (requests.incrementAndGet() <= minimumIdleSize) {
- createConnection(checkFreezed, requests, entry, initPromise, minimumIdleSize, initializedConnections);
- }
- }
- }
- });
- }
- });
-
- }
-
-
- public RFuture get(RedisCommand> command) {
- for (int j = entries.size() - 1; j >= 0; j--) {
- final ClientConnectionsEntry entry = getEntry();
- if (!entry.isFreezed()
- && tryAcquireConnection(entry)) {
- return acquireConnection(command, entry);
- }
- }
-
- List failedAttempts = new LinkedList();
- List freezed = new LinkedList();
- for (ClientConnectionsEntry entry : entries) {
- if (entry.isFreezed()) {
- freezed.add(entry.getClient().getAddr());
- } else {
- failedAttempts.add(entry.getClient().getAddr());
- }
- }
-
- StringBuilder errorMsg = new StringBuilder(getClass().getSimpleName() + " no available Redis entries. ");
- if (!freezed.isEmpty()) {
- errorMsg.append(" Disconnected hosts: " + freezed);
- }
- if (!failedAttempts.isEmpty()) {
- errorMsg.append(" Hosts disconnected due to `failedAttempts` limit reached: " + failedAttempts);
- }
-
- RedisConnectionException exception = new RedisConnectionException(errorMsg.toString());
- return connectionManager.newFailedFuture(exception);
- }
-
-
- public RFuture get(RedisCommand> command, ClientConnectionsEntry entry) {
- if ((!entry.isFreezed() || entry.getFreezeReason() == FreezeReason.SYSTEM) &&
- tryAcquireConnection(entry)) {
- return acquireConnection(command, entry);
- }
-
- RedisConnectionException exception = new RedisConnectionException(
- "Can't aquire connection to " + entry);
- return connectionManager.newFailedFuture(exception);
- }
-
-
- private void ping(RedisConnection c, final FutureListener pingListener) {
- RFuture f = c.async(RedisCommands.PING);
- f.addListener(pingListener);
- }
-
-
- public void returnConnection(ClientConnectionsEntry entry, T connection) {
- if (entry.isFreezed()) {
- connection.closeAsync();
- } else {
- releaseConnection(entry, connection);
- }
- releaseConnection(entry);
- }
-
-
- protected void releaseConnection(ClientConnectionsEntry entry) {
- entry.releaseConnection();
- }
-
-
- protected void releaseConnection(ClientConnectionsEntry entry, T conn) {
- entry.releaseConnection(conn);
- }
- }
用一张图来解释ConnectionPool干了些啥,如下图:
都到这里了,不介意再送一张图了解各种部署方式下的连接池分布了,如下图:
4.Redisson的读写操作句柄类RedissonObject
对于Redisson的任何操作,都需要获取到操作句柄类RedissonObject,RedissonObject根据不同的数据类型有不同的RedissonObject实现类,RedissonObject的类继承关系图如下:
例如想设置redis服务端的key=key的值value=123,你需要查询Redis命令和Redisson对象匹配列表,找到如下对应关系:
然后我们就知道调用代码这么写:
- Config config = new Config();
- config.useMasterSlaveServers()
- .setMasterAddress("redis://192.168.29.24:6379") // 设置redis主节点
- .addSlaveAddress("redis://192.168.29.24:7000") // 设置redis从节点
- .addSlaveAddress("redis://192.168.29.24:7001"); // 设置redis从节点
- RedissonClient redisson = Redisson.create(config);
-
-
-
- RBucket rBucket = redissonClient.getBucket("key");
-
-
- rBucket.get();
-
-
- rBucket.set("123");
至于其它的redis命令对应的redisson操作对象,都可以官网的Redis命令和Redisson对象匹配列表 查到。
6.Redisson的读写操作源码分析
从一个读操作的代码作为入口分析代码,如下:
-
- RBucket rBucket = redissonClient.getBucket("key");
-
-
- rBucket.get();
继续追踪上面RBucket的get方法,如下:
上面我们看到不管是读操作还是写操作都转交
CommandAsyncExecutor进行处理,那么这里我们需要看一下
CommandAsyncExecutor.java里关于读写操作处理的核心代码,注释代码如下:
- private NodeSource getNodeSource(String key) {
-
-
- int slot = connectionManager.calcSlot(key);
-
- MasterSlaveEntry entry = connectionManager.getEntry(slot);
-
- return new NodeSource(entry);
- }
- @Override
- public RFuture readAsync(String key, Codec codec, RedisCommand command, Object... params) {
- RPromise mainPromise = connectionManager.newPromise();
-
- NodeSource source = getNodeSource(key);
- 调用异步执行方法async
- async(true, source, codec, command, params, mainPromise, 0);
- return mainPromise;
- }
- protected void async(final boolean readOnlyMode, final NodeSource source, final Codec codec,
- final RedisCommand command, final Object[] params, final RPromise mainPromise, final int attempt) {
-
- if (mainPromise.isCancelled()) {
- free(params);
- return;
- }
-
- if (!connectionManager.getShutdownLatch().acquire()) {
- free(params);
- mainPromise.tryFailure(new RedissonShutdownException("Redisson is shutdown"));
- return;
- }
-
- final AsyncDetails details = AsyncDetails.acquire();
- if (isRedissonReferenceSupportEnabled()) {
- try {
- for (int i = 0; i < params.length; i++) {
- RedissonReference reference = RedissonObjectFactory.toReference(getConnectionManager().getCfg(), params[i]);
- if (reference != null) {
- params[i] = reference;
- }
- }
- } catch (Exception e) {
- connectionManager.getShutdownLatch().release();
- free(params);
- mainPromise.tryFailure(e);
- return;
- }
- }
-
-
-
- final RFuture connectionFuture;
- if (readOnlyMode) {
- connectionFuture = connectionManager.connectionReadOp(source, command);
- } else {
- connectionFuture = connectionManager.connectionWriteOp(source, command);
- }
-
-
- final RPromise attemptPromise = connectionManager.newPromise();
- details.init(connectionFuture, attemptPromise, readOnlyMode, source, codec, command, params, mainPromise, attempt);
-
- FutureListener mainPromiseListener = new FutureListener() {
- @Override
- public void operationComplete(Future future) throws Exception {
- if (future.isCancelled() && connectionFuture.cancel(false)) {
- log.debug("Connection obtaining canceled for {}", command);
- details.getTimeout().cancel();
- if (details.getAttemptPromise().cancel(false)) {
- free(params);
- }
- }
- }
- };
-
-
- final TimerTask retryTimerTask = new TimerTask() {
- @Override
- public void run(Timeout t) throws Exception {
- if (details.getAttemptPromise().isDone()) {
- return;
- }
- if (details.getConnectionFuture().cancel(false)) {
- connectionManager.getShutdownLatch().release();
- } else {
- if (details.getConnectionFuture().isSuccess()) {
- if (details.getWriteFuture() == null || !details.getWriteFuture().isDone()) {
- if (details.getAttempt() == connectionManager.getConfig().getRetryAttempts()) {
- if (details.getWriteFuture().cancel(false)) {
- if (details.getException() == null) {
- details.setException(new RedisTimeoutException("Unable to send command: " + command + " with params: " + LogHelper.toString(details.getParams()) + " after " + connectionManager.getConfig().getRetryAttempts() + " retry attempts"));
- }
- details.getAttemptPromise().tryFailure(details.getException());
- }
- return;
- }
- details.incAttempt();
- Timeout timeout = connectionManager.newTimeout(this, connectionManager.getConfig().getRetryInterval(), TimeUnit.MILLISECONDS);
- details.setTimeout(timeout);
- return;
- }
-
- if (details.getWriteFuture().isDone() && details.getWriteFuture().isSuccess()) {
- return;
- }
- }
- }
- if (details.getMainPromise().isCancelled()) {
- if (details.getAttemptPromise().cancel(false)) {
- free(details);
- AsyncDetails.release(details);
- }
- return;
- }
- if (details.getAttempt() == connectionManager.getConfig().getRetryAttempts()) {
- if (details.getException() == null) {
- details.setException(new RedisTimeoutException("Unable to send command: " + command + " with params: " + LogHelper.toString(details.getParams() + " after " + connectionManager.getConfig().getRetryAttempts() + " retry attempts")));
- }
- details.getAttemptPromise().tryFailure(details.getException());
- return;
- }
- if (!details.getAttemptPromise().cancel(false)) {
- return;
- }
- int count = details.getAttempt() + 1;
- if (log.isDebugEnabled()) {
- log.debug("attempt {} for command {} and params {}",
- count, details.getCommand(), Arrays.toString(details.getParams()));
- }
- details.removeMainPromiseListener();
- async(details.isReadOnlyMode(), details.getSource(), details.getCodec(), details.getCommand(), details.getParams(), details.getMainPromise(), count);
- AsyncDetails.release(details);
- }
- };
-
-
- Timeout timeout = connectionManager.newTimeout(retryTimerTask, connectionManager.getConfig().getRetryInterval(), TimeUnit.MILLISECONDS);
- details.setTimeout(timeout);
- details.setupMainPromiseListener(mainPromiseListener);
-
-
- connectionFuture.addListener(new FutureListener() {
- @Override
- public void operationComplete(Future connFuture) throws Exception {
- if (connFuture.isCancelled()) {
- return;
- }
-
- if (!connFuture.isSuccess()) {
- connectionManager.getShutdownLatch().release();
- details.setException(convertException(connectionFuture));
- return;
- }
-
- if (details.getAttemptPromise().isDone() || details.getMainPromise().isDone()) {
- releaseConnection(source, connectionFuture, details.isReadOnlyMode(), details.getAttemptPromise(), details);
- return;
- }
-
-
- final RedisConnection connection = connFuture.getNow();
-
-
- if (details.getSource().getRedirect() == Redirect.ASK) {
- List> list = new ArrayList>(2);
- RPromise promise = connectionManager.newPromise();
- list.add(new CommandData(promise, details.getCodec(), RedisCommands.ASKING, new Object[]{}));
- list.add(new CommandData(details.getAttemptPromise(), details.getCodec(), details.getCommand(), details.getParams()));
- RPromise main = connectionManager.newPromise();
- ChannelFuture future = connection.send(new CommandsData(main, list));
- details.setWriteFuture(future);
- } else {
- if (log.isDebugEnabled()) {
- log.debug("acquired connection for command {} and params {} from slot {} using node {}... {}",
- details.getCommand(), Arrays.toString(details.getParams()), details.getSource(), connection.getRedisClient().getAddr(), connection);
- }
-
- ChannelFuture future = connection.send(new CommandData(details.getAttemptPromise(), details.getCodec(), details.getCommand(), details.getParams()));
- details.setWriteFuture(future);
- }
-
- details.getWriteFuture().addListener(new ChannelFutureListener() {
- @Override
- public void operationComplete(ChannelFuture future) throws Exception {
- checkWriteFuture(details, connection);
- }
- });
-
- releaseConnection(source, connectionFuture, details.isReadOnlyMode(), details.getAttemptPromise(), details);
- }
- });
-
- attemptPromise.addListener(new FutureListener() {
- @Override
- public void operationComplete(Future future) throws Exception {
- checkAttemptFuture(source, details, future);
- }
- });
- }
上面的代码我用一张读写操作处理流程图总结如下:
至此,关于读写操作的源码讲解完毕。在上面的代码注释中,列出如下重点。
6.1 分片SLOT的计算公式
SLOT=CRC16.crc16(key.getBytes()) % MAX_SLOT
6.2 每个ConnectionPool持有的ClientConnectionsEntry对象冻结判断条件
一个节点被判断为冻结,必须同时满足以下条件:
- 该节点有slave节点,并且从节点个数大于0;
- 设置的配置ReadMode不为并且SubscriptionMode不为MASTER;
- 该节点的从节点至少有一个存活着,也即如果有从节点宕机,宕机的从节点的个数小于该节点总的从节点个数
6.3 读写负载图
7.Redisson的读写操作从连接池获取连接对象源码分析和Redisson里RedisClient使用netty源码分析
读写操作首先都需要获取到一个连接对象,在上面的分析中我们知道读写操作都是通过CommandAsyncExecutor.java里的如下代码获取连接对象:
-
-
- final RFuture connectionFuture;
- if (readOnlyMode) {
- connectionFuture = connectionManager.connectionReadOp(source, command);
- } else {
- connectionFuture = connectionManager.connectionWriteOp(source, command);
- }
上面读操作调用了
connectionManager.connectionReadOp从连接池获取连接对象,写操作调用了
connectionManager.connectionWriteOp从连接池获取连接对象,我们继续跟进
connectionManager关于connectionReadOp和connectionWriteOp的源代码,注释如下:
-
-
-
-
-
-
- public RFuture connectionReadOp(NodeSource source, RedisCommand> command) {
-
- MasterSlaveEntry entry = source.getEntry();
- if (entry == null && source.getSlot() != null) {
- entry = getEntry(source.getSlot());
- }
- if (source.getAddr() != null) {
- entry = getEntry(source.getAddr());
- if (entry == null) {
- for (MasterSlaveEntry e : getEntrySet()) {
- if (e.hasSlave(source.getAddr())) {
- entry = e;
- break;
- }
- }
- }
- if (entry == null) {
- RedisNodeNotFoundException ex = new RedisNodeNotFoundException("Node: " + source.getAddr() + " for slot: " + source.getSlot() + " hasn't been discovered yet");
- return RedissonPromise.newFailedFuture(ex);
- }
-
- return entry.connectionReadOp(command, source.getAddr());
- }
-
- if (entry == null) {
- RedisNodeNotFoundException ex = new RedisNodeNotFoundException("Node: " + source.getAddr() + " for slot: " + source.getSlot() + " hasn't been discovered yet");
- return RedissonPromise.newFailedFuture(ex);
- }
-
- return entry.connectionReadOp(command);
- }
-
-
-
-
-
-
- public RFuture connectionWriteOp(NodeSource source, RedisCommand> command) {
-
- MasterSlaveEntry entry = source.getEntry();
- if (entry == null) {
- entry = getEntry(source);
- }
- if (entry == null) {
- RedisNodeNotFoundException ex = new RedisNodeNotFoundException("Node: " + source.getAddr() + " for slot: " + source.getSlot() + " hasn't been discovered yet");
- return RedissonPromise.newFailedFuture(ex);
- }
-
- return entry.connectionWriteOp(command);
- }
我们看到上面调用
ConnectionManager从连接池获取连接对象,但是
ConnectionManager却将获取连接操作转交
MasterSlaveEntry处理,我们再一次回顾一下
MasterSlaveEntry的组成:
MasterSlaveEntry里持有中我们开篇所提到的
四个连接池,那么这里我们继续关注
MasterSlaveEntry.java的源代码:
-
-
-
-
-
- public RFuture connectionWriteOp(RedisCommand> command) {
-
-
- return writeConnectionHolder.get(command);
- }
-
-
-
-
-
-
- public RFuture connectionReadOp(RedisCommand> command) {
- if (config.getReadMode() == ReadMode.MASTER) {
-
- return connectionWriteOp(command);
- }
-
-
- return slaveBalancer.nextConnection(command);
- }
似乎又绕回来了,最终的获取连接对象都转交到了从连接池
ConnectionPool里获取连接对象,注释
ConnectionPool里的获取连接对象代码如下:
-
-
-
-
-
- public RFuture get(RedisCommand> command) {
- for (int j = entries.size() - 1; j >= 0; j--) {
- final ClientConnectionsEntry entry = getEntry();
- if (!entry.isFreezed() && tryAcquireConnection(entry)) {
-
-
-
- return acquireConnection(command, entry);
- }
- }
-
-
- List failedAttempts = new LinkedList();
- List freezed = new LinkedList();
- for (ClientConnectionsEntry entry : entries) {
- if (entry.isFreezed()) {
- freezed.add(entry.getClient().getAddr());
- } else {
- failedAttempts.add(entry.getClient().getAddr());
- }
- }
-
- StringBuilder errorMsg = new StringBuilder(getClass().getSimpleName() + " no available Redis entries. ");
- if (!freezed.isEmpty()) {
- errorMsg.append(" Disconnected hosts: " + freezed);
- }
- if (!failedAttempts.isEmpty()) {
- errorMsg.append(" Hosts disconnected due to `failedAttempts` limit reached: " + failedAttempts);
- }
-
- RedisConnectionException exception = new RedisConnectionException(errorMsg.toString());
- return connectionManager.newFailedFuture(exception);
- }
-
-
-
-
-
-
-
- private RFuture acquireConnection(RedisCommand> command, final ClientConnectionsEntry entry) {
-
- final RPromise result = connectionManager.newPromise();
-
-
- AcquireCallback callback = new AcquireCallback() {
- @Override
- public void run() {
- result.removeListener(this);
-
-
- connectTo(entry, result);
- }
-
- @Override
- public void operationComplete(Future future) throws Exception {
- entry.removeConnection(this);
- }
- };
-
- result.addListener(callback);
-
- acquireConnection(entry, callback);
-
- return result;
- }
-
-
-
-
-
-
- private void connectTo(ClientConnectionsEntry entry, RPromise promise) {
- if (promise.isDone()) {
- releaseConnection(entry);
- return;
- }
-
- T conn = poll(entry);
- if (conn != null) {
- if (!conn.isActive()) {
- promiseFailure(entry, promise, conn);
- return;
- }
-
- connectedSuccessful(entry, promise, conn);
- return;
- }
-
- createConnection(entry, promise);
- }
-
-
-
-
-
-
- protected T poll(ClientConnectionsEntry entry) {
- return (T) entry.pollConnection();
- }
-
-
-
-
-
-
- private void createConnection(final ClientConnectionsEntry entry, final RPromise promise) {
-
- RFuture connFuture = connect(entry);
- connFuture.addListener(new FutureListener() {
- @Override
- public void operationComplete(Future future) throws Exception {
- if (!future.isSuccess()) {
- promiseFailure(entry, promise, future.cause());
- return;
- }
-
- T conn = future.getNow();
- if (!conn.isActive()) {
- promiseFailure(entry, promise, conn);
- return;
- }
-
- connectedSuccessful(entry, promise, conn);
- }
- });
- }
ConnectionPool.java里获取读写操作的连接,是遍历ConnectionPool里维持的ClientConnectionsEntry列表,找到一非冻结的ClientConnectionsEntry,然后调用ClientConnectionsEntry里的freeConnectionsCounter尝试将值减1,如果成功,说明连接池中可以获取到连接,那么就从ClientConnectionsEntry里获取一个连接出来,如果拿不到连接,会调用ClientConnectionsEntry创建一个新连接放置到连接池中,并返回此连接,这里回顾一下
ClientConnectionsEntry的组成图:
我们继续跟进
ClientConnectionsEntry.java的源代码,注释如下:
-
-
-
- public RedisConnection pollConnection() {
- return freeConnections.poll();
- }
-
-
-
-
- public RFuture connect() {
-
- RFuture future = client.connectAsync();
- future.addListener(new FutureListener() {
- @Override
- public void operationComplete(Future future) throws Exception {
- if (!future.isSuccess()) {
- return;
- }
-
- RedisConnection conn = future.getNow();
- onConnect(conn);
- log.debug("new connection created: {}", conn);
- }
- });
- return future;
- }
上面的代码说明如果
ClientConnectionsEntry
里的
freeConnections
有空闲连接,那么直接返回该连接,如果没有那么调用
RedisClient.connectAsync创建一个新的连接
,这里我继续注释一下
RedisClient.java
的源代码如下:
- package org.redisson.client;
-
- import java.net.InetSocketAddress;
- import java.net.URI;
- import java.util.concurrent.ExecutorService;
- import java.util.concurrent.Executors;
- import java.util.concurrent.TimeUnit;
- import org.redisson.api.RFuture;
- import org.redisson.client.handler.RedisChannelInitializer;
- import org.redisson.client.handler.RedisChannelInitializer.Type;
- import org.redisson.misc.RPromise;
- import org.redisson.misc.RedissonPromise;
- import io.netty.bootstrap.Bootstrap;
- import io.netty.channel.Channel;
- import io.netty.channel.ChannelFuture;
- import io.netty.channel.ChannelFutureListener;
- import io.netty.channel.ChannelOption;
- import io.netty.channel.EventLoopGroup;
- import io.netty.channel.group.ChannelGroup;
- import io.netty.channel.group.ChannelGroupFuture;
- import io.netty.channel.group.DefaultChannelGroup;
- import io.netty.channel.nio.NioEventLoopGroup;
- import io.netty.channel.socket.SocketChannel;
- import io.netty.channel.socket.nio.NioSocketChannel;
- import io.netty.util.HashedWheelTimer;
- import io.netty.util.Timer;
- import io.netty.util.concurrent.Future;
- import io.netty.util.concurrent.FutureListener;
- import io.netty.util.concurrent.GlobalEventExecutor;
- import org.redisson.misc.URIBuilder;
-
-
-
-
-
- public class RedisClient {
- private final Bootstrap bootstrap;
- private final Bootstrap pubSubBootstrap;
- private final InetSocketAddress addr;
-
-
- private final ChannelGroup channels = new DefaultChannelGroup(GlobalEventExecutor.INSTANCE);
-
- private ExecutorService executor;
- private final long commandTimeout;
- private Timer timer;
- private boolean hasOwnGroup;
- private RedisClientConfig config;
-
-
- public static RedisClient create(RedisClientConfig config) {
- if (config.getTimer() == null) {
- config.setTimer(new HashedWheelTimer());
- }
- return new RedisClient(config);
- }
-
- private RedisClient(RedisClientConfig config) {
- this.config = config;
- this.executor = config.getExecutor();
- this.timer = config.getTimer();
-
- addr = new InetSocketAddress(config.getAddress().getHost(), config.getAddress().getPort());
-
- bootstrap = createBootstrap(config, Type.PLAIN);
- pubSubBootstrap = createBootstrap(config, Type.PUBSUB);
-
- this.commandTimeout = config.getCommandTimeout();
- }
-
-
- private Bootstrap createBootstrap(RedisClientConfig config, Type type) {
- Bootstrap bootstrap = new Bootstrap()
- .channel(config.getSocketChannelClass())
- .group(config.getGroup())
- .remoteAddress(addr);
-
- bootstrap.handler(new RedisChannelInitializer(bootstrap, config, this, channels, type));
-
- bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, config.getConnectTimeout());
- return bootstrap;
- }
-
-
- @Deprecated
- public RedisClient(String address) {
- this(URIBuilder.create(address));
- }
-
-
- @Deprecated
- public RedisClient(URI address) {
- this(new HashedWheelTimer(), Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() * 2), new NioEventLoopGroup(), address);
- hasOwnGroup = true;
- }
-
-
- @Deprecated
- public RedisClient(Timer timer, ExecutorService executor, EventLoopGroup group, URI address) {
- this(timer, executor, group, address.getHost(), address.getPort());
- }
-
-
- @Deprecated
- public RedisClient(String host, int port) {
- this(new HashedWheelTimer(), Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() * 2), new NioEventLoopGroup(), NioSocketChannel.class, host, port, 10000, 10000);
- hasOwnGroup = true;
- }
-
-
- @Deprecated
- public RedisClient(Timer timer, ExecutorService executor, EventLoopGroup group, String host, int port) {
- this(timer, executor, group, NioSocketChannel.class, host, port, 10000, 10000);
- }
-
-
- @Deprecated
- public RedisClient(String host, int port, int connectTimeout, int commandTimeout) {
- this(new HashedWheelTimer(), Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() * 2), new NioEventLoopGroup(), NioSocketChannel.class, host, port, connectTimeout, commandTimeout);
- }
-
-
- @Deprecated
- public RedisClient(final Timer timer, ExecutorService executor, EventLoopGroup group, Class extends SocketChannel> socketChannelClass, String host, int port,
- int connectTimeout, int commandTimeout) {
- RedisClientConfig config = new RedisClientConfig();
- config.setTimer(timer).setExecutor(executor).setGroup(group).setSocketChannelClass(socketChannelClass)
- .setAddress(host, port).setConnectTimeout(connectTimeout).setCommandTimeout(commandTimeout);
-
- this.config = config;
- this.executor = config.getExecutor();
- this.timer = config.getTimer();
-
- addr = new InetSocketAddress(config.getAddress().getHost(), config.getAddress().getPort());
-
-
- bootstrap = createBootstrap(config, Type.PLAIN);
- pubSubBootstrap = createBootstrap(config, Type.PUBSUB);
-
- this.commandTimeout = config.getCommandTimeout();
- }
-
-
- public String getIpAddr() {
- return addr.getAddress().getHostAddress() + ":" + addr.getPort();
- }
-
- public InetSocketAddress getAddr() {
- return addr;
- }
-
- public long getCommandTimeout() {
- return commandTimeout;
- }
-
- public EventLoopGroup getEventLoopGroup() {
- return bootstrap.config().group();
- }
-
- public RedisClientConfig getConfig() {
- return config;
- }
-
- public RedisConnection connect() {
- try {
- return connectAsync().syncUninterruptibly().getNow();
- } catch (Exception e) {
- throw new RedisConnectionException("Unable to connect to: " + addr, e);
- }
- }
-
- public RFuture connectAsync() {
- final RPromise f = new RedissonPromise();
-
- ChannelFuture channelFuture = bootstrap.connect();
- channelFuture.addListener(new ChannelFutureListener() {
- @Override
- public void operationComplete(final ChannelFuture future) throws Exception {
- if (future.isSuccess()) {
-
- final RedisConnection c = RedisConnection.getFrom(future.channel());
- c.getConnectionPromise().addListener(new FutureListener() {
- @Override
- public void operationComplete(final Future future) throws Exception {
- bootstrap.config().group().execute(new Runnable() {
- @Override
- public void run() {
- if (future.isSuccess()) {
- if (!f.trySuccess(c)) {
- c.closeAsync();
- }
- } else {
- f.tryFailure(future.cause());
- c.closeAsync();
- }
- }
- });
- }
- });
- } else {
- bootstrap.config().group().execute(new Runnable() {
- public void run() {
- f.tryFailure(future.cause());
- }
- });
- }
- }
- });
- return f;
- }
-
- public RedisPubSubConnection connectPubSub() {
- try {
- return connectPubSubAsync().syncUninterruptibly().getNow();
- } catch (Exception e) {
- throw new RedisConnectionException("Unable to connect to: " + addr, e);
- }
- }
-
-
- public RFuture connectPubSubAsync() {
- final RPromise f = new RedissonPromise();
-
- ChannelFuture channelFuture = pubSubBootstrap.connect();
- channelFuture.addListener(new ChannelFutureListener() {
- @Override
- public void operationComplete(final ChannelFuture future) throws Exception {
- if (future.isSuccess()) {
-
- final RedisPubSubConnection c = RedisPubSubConnection.getFrom(future.channel());
- c.getConnectionPromise().addListener(new FutureListener() {
- @Override
- public void operationComplete(final Future future) throws Exception {
- bootstrap.config().group().execute(new Runnable() {
- @Override
- public void run() {
- if (future.isSuccess()) {
- if (!f.trySuccess(c)) {
- c.closeAsync();
- }
- } else {
- f.tryFailure(future.cause());
- c.closeAsync();
- }
- }
- });
- }
- });
- } else {
- bootstrap.config().group().execute(new Runnable() {
- public void run() {
- f.tryFailure(future.cause());
- }
- });
- }
- }
- });
- return f;
- }
-
-
- public void shutdown() {
- shutdownAsync().syncUninterruptibly();
- if (hasOwnGroup) {
- timer.stop();
- executor.shutdown();
- try {
- executor.awaitTermination(15, TimeUnit.SECONDS);
- } catch (InterruptedException e) {
- Thread.currentThread().interrupt();
- }
- bootstrap.config().group().shutdownGracefully();
-
- }
- }
-
-
- public ChannelGroupFuture shutdownAsync() {
- for (Channel channel : channels) {
- RedisConnection connection = RedisConnection.getFrom(channel);
- if (connection != null) {
- connection.setClosed(true);
- }
- }
- return channels.close();
- }
-
- @Override
- public String toString() {
- return "[addr=" + addr + "]";
- }
- }
上面就是Redisson利用java网络编程框架netty连接redis的全过程
,如果你对
netty
比较熟悉,阅读上面的代码应该不是问题。
转载地址:http://aperise.iteye.com/blog/2400528