分布式锁基于Redis和Zookeeper的实现方案

一,为什么要用分布式锁?

如果不同的系统之间共享了一组资源,那么访问这组资源的时候,往往就需要通过一些互斥的手段来防止彼此间的干扰,以保证数据的一致性。

如图1所示,在单机部署的系统中,使用线程锁来解决高并发的问题,多线程访问共享变量的问题达到数据一致性,例如使用synchornized,的的ReentrantLock等。

2,在多机部署的系统中,是在不同的JVM虚拟机中运行的,就无法使用JDK提供的阿比来解决并发的问题,这个时候就需要使用分布式锁来应对。


对于分布式的实现方案,目前有几种比较流行的做法:

1,基于Redis的的实现分布式锁

2,基于数据库实现分布式锁

3,基于动物园管理员实现分布式锁

这里主要讲解基于Redis的的,动物园管理员的实现方案


二,基于Redis的的的实现方案

2.1,单节点的实现方案

2.1.1,获取锁

客户端向Redis的的节点发送如下命令:

SET resource_name my_random_value NX PX 30000

RESOURCE_NAME:关键值

my_random_value:这个必须是唯一的随机字符串。

NX:表示键必须不存在,才可以设置成功,用于添加;这保证了只有第一个请求的客户端才能获得锁,而其它客户端在锁被释放之前都无法获得锁。

PX 30000:表示为键设置毫秒级的过期时间,这个锁有一个30秒的自动过期时间。这里可以根据业务场景选择合适的过期时间。


这里有两点需要注意的是:

1,这里my_random_value为啥必须是唯一的随机字符串,而不是一个固定的值考虑一种情况?

如图1所示,客户端1首先获取到了锁

2,客户端1长时间的GC暂停

3,客户端1的过期时间到了释放了锁

如图4所示,此时客户端2套成功获取到同一资源的锁

5,客户端1从从阻塞种恢复过来,执行业务逻辑完成后,释放了客户端的锁

上面的这种情况明显的体现了my_random_value不能设置为一个固定值,因为这样客户端在访问共享资源时,无法保证锁可以提供正确的访问保证


2,将获取锁的命令分成两步操作

Long result = jedis.setnx("key", "value");
if (result == 1) {
   jedis.expire("key", 3000);
}

为什么说这里会有问题呢?

如图1所示,客户端1执行SETNX成功后崩溃了,后续也就谈不上执行到期设置超时时间了,就会导致客户端1一直持有锁

2,setnx和expire虽然可以起到和SET resource_name my_random_value NX PX 30000相同的作用,但却不是原子的


2.1.2:释放锁

执行下面的Redis Lua脚本来释放锁,Lua脚本在Redis中是原子执行的,执行过程中间不会插入其他命令

if redis.call("get",KEYS[1]) == ARGV[1] then
   return redis.call("del",KEYS[1])
else
   return 0
end

KEYS [1]:RESOURCE_NAME作为KEYS [1]的值传入

ARGV [1]:my_random_value作为ARGV [1]的值传入


到这里基于单节点的redis的的分布式锁就结束了,那么问题来了,如果这个节点挂了,服务也就不可用了,客户端也就无法获取到锁了,此时就要解决不高可用的问题,就需要部署redis集群,集群的部署可以参见:Redis三主三从集群搭建


下面提供一个完整的获取,释放锁的代码

public class RedisLock {

   private static final String ACQUIRE_SUCCESS="ok";

   private static final Long RELEASE_SUCCESS = 1L;

   private static final String KET_NOT_EXSIT="NX";

   private static final String KET_EXPIRE_TIME="PX";

   private JedisCluster jedis;


   /**获取锁
    *
    * @param key
    * @param randomValue
    * @param expireTime
    * @return
    */

   public boolean acquireLock(String key,String randomValue,long expireTime){

       String result = jedis.set(key, randomValue, KET_NOT_EXSIT, KET_EXPIRE_TIME, expireTime);
       if (ACQUIRE_SUCCESS.equals(result)){
           return true;
       }
       return false;

   }

   /**
    * 释放锁
    * @param key
    * @param randomValue
    * @return
    */

   public boolean releaseLock(String key,String randomValue){

       String script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end";
       Object result = jedis.eval(script, Collections.singletonList(key), Collections.singletonList(randomValue));
       if (RELEASE_SUCCESS.equals(result)){
           return true;
       }
       return false;
   }
}


其实这里还是有问题的,参考下面的场景:

如图1所示,客户端1执行acquireLock获取到锁。

2,客户端1因为某种原因导致长时间的GC暂停,这个时间大于锁的过期时间

3,此时客户端2执行acquireLock获取到了相同的锁。

如图4所示,客户端1从长时间的GC暂停中恢复过来,此时它并不知道自己持有的锁过期了它依然向共享资源发起了写数据请求。

5,这时锁实际上被客户端2持有,因此两个客户端的写请求就有可能冲突,锁的互斥作用失效了。


redis的的作者antirez提出了新的分布式算法Redlock,上述说的GC暂停的问题都可以在这里解决,但是会存在时钟跳跃的问题会导致分布式锁不安全的问题。关于时钟问题,Redis的分布式锁是否安全的问题,大家可以搜索微信公众号:铁勒,博客,查看张铁蕾蕾哥的基于Redis的的分布式锁到底安全吗的博文有详细解释,相当好!


三,基于动物园管理员的分布式锁实现方案

这里主要讲基于的ZooKeeper的实现排他锁,共享锁


3.1,排他锁

假设我们需要给你一个商品加锁,基本思路如下:

1,创建ZK的一个临时节点,模拟给某个商品ID加锁。

2,ZK会保证只会创建一个临时节点,其他请求过来如果再要创建临时节点,就会NodeExistsException异常。

3,如果临时节点创建成功了,那么说明我们成功加锁了,此时就可以去执行对应的业务逻辑。

如图4所示,如果临时节点创建失败了,说明有人已经在拿到锁了,那么就不断的等待,直到自己可以获取到锁为止。

5,释放一个分布式锁,删除掉那个临时节点就可以了,就代表释放了一个锁,那么此时其他的机器就可以成功创建临时节点,获取到锁。

这种分布式锁的做法比较简单但是很实用,可以满足大部分的业务场景,代码如下:

public class ZooKeeperSession {

   private static CountDownLatch connectedSemaphore = new CountDownLatch(1);

   private ZooKeeper zookeeper;

   public ZooKeeperSession() {
       // 连接zookeeper server,创建会话的时候,是异步去进行的,所以要给一个监听器告诉我们何时才真正完成了跟zk server的连接
       try {
           this.zookeeper = new ZooKeeper(
                   "192.168.31.193:2181,192.168.31.160:2181,192.168.31.114:2181",
                   40000,
                   new ZooKeeperWatcher());
           // 给一个状态CONNECTING,连接中
           System.out.println(zookeeper.getState());

           try {

               // 如果数字减到0,那么之前所有在await的线程,都会逃出阻塞的状 继续向下运行
               connectedSemaphore.await();
           } catch(InterruptedException e) {
               e.printStackTrace();
           }

           System.out.println("ZooKeeper session established......");
       } catch (Exception e) {
           e.printStackTrace();
       }
   }

   /**
    * 获取分布式锁
    * @param productId
    */

   public void acquireDistributedLock(Long productId) {
       String path = "/product-lock-" + productId;

       try {
           zookeeper.create(path, "".getBytes(),
                   Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
           System.out.println("success to acquire lock for product[id=" + productId + "]");
       } catch (Exception e) {
           // 如果商品对应的锁的node已经存在了,那么就会报异常,说明锁已经被其他线程获取了
           int count = 0;
           while(true) {
               try {
                   Thread.sleep(20);
                   zookeeper.create(path, "".getBytes(),
                           Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
               } catch (Exception e2) {
                   e2.printStackTrace();
                   count++;
                   continue;
               }
               System.out.println("success to acquire lock for product[id=" + productId + "] after " + count + " times try......");
               break;
           }
       }
   }

   /**
    * 释放掉一个分布式锁
    * @param productId
    */

   public void releaseDistributedLock(Long productId) {
       String path = "/product-lock-" + productId;
       try {
           zookeeper.delete(path, -1);
       } catch (Exception e) {
           e.printStackTrace();
       }
   }

   /**
    * 建立zk session的watcher
    *
    */

   private class ZooKeeperWatcher implements Watcher {

       public void process(WatchedEvent event) {
           System.out.println("Receive watched event: " + event.getState());
           if(KeeperState.SyncConnected == event.getState()) {
               connectedSemaphore.countDown();
           }
       }

   }

   /**
    * 封装单例的静态内部类
    *
    */

   private static class Singleton {

       private static ZooKeeperSession instance;

       static {
           instance = new ZooKeeperSession();
       }

       public static ZooKeeperSession getInstance() {
           return instance;
       }

   }

   /**
    * 获取单例
    * @return
    */

   public static ZooKeeperSession getInstance() {
       return Singleton.getInstance();
   }

   /**
    * 初始化单例的便捷方法
    */

   public static void init() {
       getInstance();
   }
}


3.2,共享锁

。动物园管理员提供的临时顺序节点节点就可以实现分布式共享锁对于共享锁:

1,读请求:如果没有比自己序号小的子节点,或是所有比自己序号小的子节点都是读请求,那么表明可以成功获取到共享锁;如果比自己序号小的子节点中有写请求,则进入等待。

2,写请求:如果自己不是序号最小的子节点,那么就需要进入等待。


大致的思路如下:

如图1所示,根据业务场景对于读的场景的路径前缀为:product_read_,写场景的路径前缀为product_wirte_。

2,创建一个临时顺序节点。

3,调用的的getChildren()接口来获取所有已经创建的子节点列表。

如图4所示,判断当前创建的节点是否为第一个节点,如果是则获取分布式锁,返回真。

5,如果第四步不成立,则对于读请求:所有比自己序号小的子节点都是读请求,那么表明可以成功获取到共享锁;如果比自己序号小的子节点中有写请求,则进入等待对于写请求:自己不是序号最小的子节点,那么就需要进入等待。

如图6所示,如果无法获取共享锁则调用存在()来对比自己小的节点注册观察程序读请求:向比自己序号小的最后一个写请求节点注册观察者监听,写请求向比自己序号小的最后一个节点观察家注册。

如图7所示,等待观察者通知,再次进入到第3步。


代码实现

public class ZookeeperLock {


   private static Logger LOGGER = LoggerFactory.getLogger(ZookeeperLock.class);

   /**
    * 前缀都为/lock,
    * 获取共享锁用于写请求:/product_write_ ,这是可以根据业务需要动态的传入,写在这里是为了方便阅读代码
    * 获取共享锁用于读请求:/product_read_
    */

   private String PREFIX_LOCK_PATH = "/lock";
   private String WIRITE_LOCK_PATH = "product_write_";
   private String READ_LOCK_PATH = "/product_read_";

   //监听器
   private DefaultWatcher watcher;

   private ZooKeeper zookeeper;

   //连接zk server
   public ZookeeperLock() {
       try {
           LOGGER.info("trying connect ZooKeeper Server ......");
           this.zookeeper = new ZooKeeper(
                   "192.168.31.193:2181,192.168.31.162:2181,192.168.31.114:2181",
                   4000, new Watcher() {
               @Override
               public void process(WatchedEvent event) {
                   LOGGER.info(" receive event : " + event.getType().name());
               }
           });
           LOGGER.info("ZooKeeper Server has connected,and state=[" + zookeeper.getState() + "]");
       } catch (Exception e) {
           e.printStackTrace();
       }
   }

   /**
    * 封装单例的静态内部类
    */

   private static class Singleton {

       private static volatile ZookeeperLock instance;

       static {
           instance = new ZookeeperLock();
       }

       public static ZookeeperLock getInstance() {
           return instance;
       }

   }


   /**
    * 获取单例
    *
    * @return
    */

   public static ZookeeperLock getInstance() {
       return Singleton.getInstance();
   }

   /**
    * 初始化单例方法
    */

   public static void init() {
       getInstance();
   }


   /**
    * 获取锁:创建一个临时顺序节点
    *
    * @param path :WIRITE_LOCK_PATH:/product_write_,READ_LOCK_PATH:/product_read_ 需要根据读写场景传入
    * @return
    */

   public String acquireDistributedLock(String path, Long prductId) {
       try {
           path = PREFIX_LOCK_PATH + path + prductId;
           String currentPath = zookeeper.create(path, "".getBytes(), Ids.OPEN_ACL_UNSAFE,
                   CreateMode.EPHEMERAL_SEQUENTIAL);
           acquireSharedLock(currentPath);
           return currentPath;
       } catch (Exception e) {
           e.printStackTrace();
       }
       return null;
   }

   /**
    * 处理获取共享锁的具体逻辑
    *
    * @param currentPath
    * @return
    * @throws KeeperException
    * @throws InterruptedException
    */

   private boolean acquireSharedLock(String currentPath) throws KeeperException, InterruptedException {
       // 获取所有已经创建的子节点列表
       List childrenNodes = zookeeper.getChildren(PREFIX_LOCK_PATH, false);
       //最小的id先拿到锁,比较自己本身是否为最小的id
       Collections.sort(childrenNodes);
       int currentNodeIndex = childrenNodes.indexOf(currentPath.substring(PREFIX_LOCK_PATH.length() + 1));
       //如果当前创建的为第一个节点则说明可以立即获取分布式锁
       if (currentNodeIndex == 0) {
           LOGGER.info(" acquire distribute lock, lock path: " + currentPath);
           return true;
       } else {
           /**
            *  1、对于读请求:如果没有比自己序号小的子节点或者所有比自己小的子节点都是读请求,那么可以获取共享锁
            *  如果存在比自己序号小的子节点中有写请求,那么就需要等待
            *  2、因此从小于index开始判断
            *  3、countWirteLock用于记录写锁的数量
            *  4、如果countWirteLock==0说明比自己小的序号全部为读锁,则当前可以获取共享锁
            */

           int countWirteLock = 0;
           for (int i = 0; i < currentNodeIndex; i++) {
               if (childrenNodes.get(i).contains(WIRITE_LOCK_PATH)) {
                   countWirteLock++;
                   if (countWirteLock >= 1) break;
               }
           }
           if (countWirteLock == 0 && !currentPath.contains(WIRITE_LOCK_PATH)) {
               return true;
           }
           byte[] mutex = new byte[0];
           watcher = new DefaultWatcher(mutex);
           // 查询前一个目录是否存在,并且注册目录事件监听器,监听一次事件后即删除
           String preLockPath = childrenNodes.get(currentNodeIndex - 1);
           Stat state = zookeeper.exists(PREFIX_LOCK_PATH + "/" + preLockPath, watcher);
           // 如果state为null则重新尝试获取共享锁,如果不为空则等待
           if (state == null) {
               return acquireSharedLock(currentPath);
           } else {
               LOGGER.info("waitting for the pre path = [" + preLockPath + "]");
               synchronized (mutex) {
                   // 等待删除目录事件唤醒
                   mutex.wait();
               }
               return acquireSharedLock(currentPath);
           }
       }

   }


   /**
    * 监控器类
    */

   public static class DefaultWatcher implements Watcher {
       private byte[] mutex;

       public DefaultWatcher(byte[] mutex) {
           this.mutex = mutex;
       }

       public void process(WatchedEvent event) {
           synchronized (mutex) {
               mutex.notifyAll();
           }
       }
   }

   /**
    * 分布式锁的释放
    *
    * @param path
    */

   public void releaseDistributedLock(String path) {
       try {
           zookeeper.delete(path, -1);
           System.out.println("Release lock, lock path is" + path);
       } catch (Exception e) {
           e.printStackTrace();
       }
   }


}


测试,这里使用多线程模拟

public class ZooKeeperTest {
  private static String WIRITE_LOCK_PATH = "/product_write_";
  private static String READ_LOCK_PATH = "/product_read_";

  private static CountDownLatch connectedSemaphore = new CountDownLatch(1);
  public static void main(String[] args) {


     for (long i = 1; i <=10; i++) {
        Thread thread = new Thread(new Runnable() {
           @Override
           public void run() {
              ZookeeperLock zkSession = ZookeeperLock.getInstance();
              String path = zkSession.acquireDistributedLock(WIRITE_LOCK_PATH, 1l);
              System.out.println(Thread.currentThread().getName()+" 获取到锁,path="+path);
              try {
                 Thread.sleep(2000);
              } catch (InterruptedException e) {
                 e.printStackTrace();
              }
              zkSession.releaseDistributedLock(path);
           }
        }) {
           //如果不覆写toString的话线程名看着不太清晰
           public String toString() {
              return getName();
           }
        };
        thread.setName("thread-" + i);
        thread.start();
     }

  }


}


查看测试/ product_read_:读锁时的控制台日志

线程6先获取到了锁并且没有阻塞其他线程获取锁,其他线程在读请求时也可以正常的获取锁。

thread-6 获取到锁,path=/lock/product_read_10000000449
13:10:59.376 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 12,8  replyHeader:: 12,4294968149,0  request:: '/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-4 获取到锁,path=/lock/product_read_10000000450
13:10:59.377 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 13,8  replyHeader:: 13,4294968149,0  request:: '
/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-10 获取到锁,path=/lock/product_read_10000000451
13:10:59.379 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 14,8  replyHeader:: 14,4294968149,0  request:: '/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-9 获取到锁,path=/lock/product_read_10000000452
13:10:59.380 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 15,8  replyHeader:: 15,4294968149,0  request:: '
/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-7 获取到锁,path=/lock/product_read_10000000453
13:10:59.381 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 16,8  replyHeader:: 16,4294968149,0  request:: '/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-2 获取到锁,path=/lock/product_read_10000000454
13:10:59.382 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 17,8  replyHeader:: 17,4294968149,0  request:: '
/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-5 获取到锁,path=/lock/product_read_10000000455
13:10:59.383 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 18,8  replyHeader:: 18,4294968149,0  request:: '/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-8 获取到锁,path=/lock/product_read_10000000456
13:10:59.384 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 19,8  replyHeader:: 19,4294968149,0  request:: '
/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-1 获取到锁,path=/lock/product_read_10000000457
13:10:59.385 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 20,8  replyHeader:: 20,4294968149,0  request:: '/lock,F  response:: v{'product_read_10000000450,'product_read_10000000451,'product_read_10000000452,'product_read_10000000453,'product_read_10000000454,'product_read_10000000455,'product_read_10000000456,'product_read_10000000457,'product_read_10000000458,'product_read_10000000449}
thread-3 获取到锁,path=/lock/product_read_10000000458


13:11:00.714 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde052001f after 1ms
13:11:01.388 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 21,2  replyHeader:: 21,4294968150,0  request:: '/lock/product_read_10000000449,-1  response:: null
Release lock, lock path is/lock/product_read_10000000449
13:11:01.389 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 22,2  replyHeader:: 22,4294968151,0  request:: '
/lock/product_read_10000000450,-1  response:: null
Release lock, lock path is/lock/product_read_10000000450
13:11:01.390 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 23,2  replyHeader:: 23,4294968152,0  request:: '/lock/product_read_10000000451,-1  response:: null
Release lock, lock path is/lock/product_read_10000000451
13:11:01.392 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 24,2  replyHeader:: 24,4294968153,0  request:: '
/lock/product_read_10000000452,-1  response:: null
Release lock, lock path is/lock/product_read_10000000452
13:11:01.393 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 25,2  replyHeader:: 25,4294968154,0  request:: '/lock/product_read_10000000453,-1  response:: null
Release lock, lock path is/lock/product_read_10000000453
13:11:01.393 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 26,2  replyHeader:: 26,4294968155,0  request:: '
/lock/product_read_10000000454,-1  response:: null
Release lock, lock path is/lock/product_read_10000000454
13:11:01.394 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 27,2  replyHeader:: 27,4294968156,0  request:: '/lock/product_read_10000000455,-1  response:: null
Release lock, lock path is/lock/product_read_10000000455
13:11:01.394 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 28,2  replyHeader:: 28,4294968157,0  request:: '
/lock/product_read_10000000456,-1  response:: null
Release lock, lock path is/lock/product_read_10000000456
13:11:01.395 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 29,2  replyHeader:: 29,4294968158,0  request:: '/lock/product_read_10000000457,-1  response:: null
Release lock, lock path is/lock/product_read_10000000457
13:11:01.395 [thread-7-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde052001f, packet:: clientPath:null serverPath:null finished:false header:: 30,2  replyHeader:: 30,4294968159,0  request:: '
/lock/product_read_10000000458,-1  response:: null
Release lock, lock path is/lock/product_read_10000000458


查看测试/ product_write_:写锁时的控制台日志

当线程1获取当写锁时,路径= /锁定/ product_write_10000000459,线程4创建节点的路径为/锁定/ product_write_10000000460并且等待459(取尾号了)的释放,当459释放后,从日志可以到螺纹 - 2才获取到锁。其他的依次类推。

13:15:45.767 [thread-1] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000459
thread-1 获取到锁,path=/lock/product_write_10000000459
13:15:45.782 [thread-4] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000459]
13:15:45.783 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 22,3  replyHeader:: 22,4294968171,0  request:: '/lock/product_write_10000000460,T  response:: s{4294968163,4294968163,1527174761277,1527174761277,0,0,0,244199118225932320,0,0,4294968163}
13:15:45.784 [thread-2] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000460]
13:15:45.784 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 23,3  replyHeader:: 23,4294968171,0  request:: '
/lock/product_write_10000000461,T  response:: s{4294968164,4294968164,1527174761278,1527174761278,0,0,0,244199118225932320,0,0,4294968164}
13:15:45.785 [thread-3] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000461]
13:15:45.785 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 24,3  replyHeader:: 24,4294968171,0  request:: '/lock/product_write_10000000462,T  response:: s{4294968165,4294968165,1527174761278,1527174761278,0,0,0,244199118225932320,0,0,4294968165}
13:15:45.785 [thread-8] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000462]
13:15:45.786 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 25,3  replyHeader:: 25,4294968171,0  request:: '
/lock/product_write_10000000463,T  response:: s{4294968166,4294968166,1527174761278,1527174761278,0,0,0,244199118225932320,0,0,4294968166}
13:15:45.786 [thread-7] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000463]
13:15:45.787 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 26,3  replyHeader:: 26,4294968171,0  request:: '/lock/product_write_10000000464,T  response:: s{4294968167,4294968167,1527174761278,1527174761278,0,0,0,244199118225932320,0,0,4294968167}
13:15:45.787 [thread-10] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000464]
13:15:45.788 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 27,3  replyHeader:: 27,4294968171,0  request:: '
/lock/product_write_10000000465,T  response:: s{4294968168,4294968168,1527174761278,1527174761278,0,0,0,244199118225932320,0,0,4294968168}
13:15:45.788 [thread-9] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000465]
13:15:45.789 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 28,3  replyHeader:: 28,4294968171,0  request:: '/lock/product_write_10000000466,T  response:: s{4294968169,4294968169,1527174761278,1527174761278,0,0,0,244199118225932320,0,0,4294968169}
13:15:45.789 [thread-6] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000466]
13:15:45.790 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 29,3  replyHeader:: 29,4294968171,0  request:: '
/lock/product_write_10000000467,T  response:: s{4294968170,4294968170,1527174761278,1527174761278,0,0,0,244199118225932320,0,0,4294968170}
13:15:45.790 [thread-5] INFO zk.ZookeeperLock - waitting for the pre path = [product_write_10000000467]


13:15:47.119 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 2ms
13:15:47.778 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:15:47.779 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000459 for sessionid 0x36391cde0520020
13:15:47.780 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 30,2  replyHeader:: 30,4294968172,0  request:: '/lock/product_write_10000000459,-1  response:: null
Release lock, lock path is/lock/product_write_10000000459
13:15:47.782 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 31,8  replyHeader:: 31,4294968172,0  request:: '
/lock,F  response:: v{'product_write_10000000460,'product_write_10000000466,'product_write_10000000465,'product_write_10000000468,'product_write_10000000467,'product_write_10000000462,'product_write_10000000461,'product_write_10000000464,'product_write_10000000463}
13:15:47.782 [thread-4] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000460
thread-4 获取到锁,path=/lock/product_write_10000000460
13:15:49.116 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 0ms
13:15:49.791 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:15:49.791 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000460 for sessionid 0x36391cde0520020
13:15:49.792 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 32,2  replyHeader:: 32,4294968173,0  request:: '
/lock/product_write_10000000460,-1  response:: null
Release lock, lock path is/lock/product_write_10000000460
13:15:49.793 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 33,8  replyHeader:: 33,4294968173,0  request:: '/lock,F  response:: v{'product_write_10000000466,'product_write_10000000465,'product_write_10000000468,'product_write_10000000467,'product_write_10000000462,'product_write_10000000461,'product_write_10000000464,'product_write_10000000463}
13:15:49.794 [thread-2] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000461
thread-2 获取到锁,path=/lock/product_write_10000000461
13:15:51.131 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 1ms
13:15:51.801 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:15:51.802 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000461 for sessionid 0x36391cde0520020
13:15:51.802 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 34,2  replyHeader:: 34,4294968174,0  request:: '
/lock/product_write_10000000461,-1  response:: null
Release lock, lock path is/lock/product_write_10000000461
13:15:51.804 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 35,8  replyHeader:: 35,4294968174,0  request:: '/lock,F  response:: v{'product_write_10000000466,'product_write_10000000465,'product_write_10000000468,'product_write_10000000467,'product_write_10000000462,'product_write_10000000464,'product_write_10000000463}
13:15:51.804 [thread-3] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000462
thread-3 获取到锁,path=/lock/product_write_10000000462
13:15:53.142 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 0ms
13:15:53.811 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:15:53.811 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000462 for sessionid 0x36391cde0520020
13:15:53.812 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 36,2  replyHeader:: 36,4294968175,0  request:: '/lock/product_write_10000000462,-1  response:: null
Release lock, lock path is/lock/product_write_10000000462
13:15:53.813 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 37,8  replyHeader:: 37,4294968175,0  request:: '
/lock,F  response:: v{'product_write_10000000466,'product_write_10000000465,'product_write_10000000468,'product_write_10000000467,'product_write_10000000464,'product_write_10000000463}
13:15:53.813 [thread-8] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000463
thread-8 获取到锁,path=/lock/product_write_10000000463
13:15:55.149 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 1ms
13:15:55.822 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:15:55.822 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000463 for sessionid 0x36391cde0520020
13:15:55.823 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 38,2  replyHeader:: 38,4294968176,0  request:: '/lock/product_write_10000000463,-1  response:: null
Release lock, lock path is/lock/product_write_10000000463
13:15:55.824 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 39,8  replyHeader:: 39,4294968176,0  request:: '
/lock,F  response:: v{'product_write_10000000466,'product_write_10000000465,'product_write_10000000468,'product_write_10000000467,'product_write_10000000464}
13:15:55.824 [thread-7] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000464
thread-7 获取到锁,path=/lock/product_write_10000000464
13:15:57.162 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 1ms
13:15:57.832 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:15:57.833 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000464 for sessionid 0x36391cde0520020
13:15:57.833 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 40,2  replyHeader:: 40,4294968177,0  request:: '
/lock/product_write_10000000464,-1  response:: null
Release lock, lock path is/lock/product_write_10000000464
13:15:57.835 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 41,8  replyHeader:: 41,4294968177,0  request:: '/lock,F  response:: v{'product_write_10000000466,'product_write_10000000465,'product_write_10000000468,'product_write_10000000467}
13:15:57.835 [thread-10] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000465
thread-10 获取到锁,path=/lock/product_write_10000000465
13:15:59.172 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 1ms
13:15:59.844 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:15:59.844 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000465 for sessionid 0x36391cde0520020
13:15:59.845 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 42,2  replyHeader:: 42,4294968178,0  request:: '
/lock/product_write_10000000465,-1  response:: null
Release lock, lock path is/lock/product_write_10000000465
13:15:59.846 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 43,8  replyHeader:: 43,4294968178,0  request:: '/lock,F  response:: v{'product_write_10000000466,'product_write_10000000468,'product_write_10000000467}
13:15:59.846 [thread-9] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000466
thread-9 获取到锁,path=/lock/product_write_10000000466
13:16:01.184 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 1ms
13:16:01.856 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:16:01.856 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000466 for sessionid 0x36391cde0520020
13:16:01.856 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 44,2  replyHeader:: 44,4294968179,0  request:: '/lock/product_write_10000000466,-1  response:: null
Release lock, lock path is/lock/product_write_10000000466
13:16:01.858 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 45,8  replyHeader:: 45,4294968179,0  request:: '
/lock,F  response:: v{'product_write_10000000468,'product_write_10000000467}
13:16:01.858 [thread-6] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000467
thread-6 获取到锁,path=/lock/product_write_10000000467
13:16:03.196 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 0ms
13:16:03.866 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got notification sessionid:0x36391cde0520020
13:16:03.866 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected type:NodeDeleted path:/lock/product_write_10000000467 for sessionid 0x36391cde0520020
13:16:03.867 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 46,2  replyHeader:: 46,4294968180,0  request:: '/lock/product_write_10000000467,-1  response:: null
Release lock, lock path is/lock/product_write_10000000467
13:16:03.868 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 47,8  replyHeader:: 47,4294968180,0  request:: '
/lock,F  response:: v{'product_write_10000000468}
13:16:03.868 [thread-5] INFO zk.ZookeeperLock -  acquire distribute lock, lock path: /lock/product_write_10000000468
thread-5 获取到锁,path=/lock/product_write_10000000468
13:16:05.206 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x36391cde0520020 after 0ms
13:16:05.876 [thread-1-SendThread(192.168.31.114:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x36391cde0520020, packet:: clientPath:null serverPath:null finished:false header:: 48,2  replyHeader:: 48,4294968181,0  request:: '
/lock/product_write_10000000468,-1  response:: null
Release lock, lock path is/lock/product_write_10000000468


四,总结

如图1所示,如果是为了效率而使用分布式锁,允许锁的偶尔失效,那么使用单Redis的的节点的锁方案就足够了,简单而且效率高。

2,如果是为了正确性而使用分布式锁,应该考虑类似动物园管理员的方案,或者支持事务的数据库。

3,基于redis的的的分布式锁除了上面说的问题,其对过期时间设置也是个问题,如果过期时间设置过短,可能业务逻辑没有执行完,就释放了锁,从而产生了安全性的问题,如果过期时间设置过长,又会导致客户端持有锁的时间过长从而影响系统性能。

4,基于动物园管理员的分布式锁锁是依靠会议(心跳)来维持锁的持有状态的,而Redis的的不支持使使sesion。

5,基于动物园管理员的锁支持在获取锁失败之后等待锁重新释放的事件,观察器的机制,而redis的的不支持。

如图6所示,这里就不去叙说数据库的实现了网上有很多资料,这里只说明下操作数据库是需要一定的开销的,很多大型分布式系统的瓶颈都是在数据库的操作上。



参考文章

付磊,张益军:“Redis的的开发与运维”

倪超:“的的Paxos到动物园管理员:分布式一致性原理与实践”

张铁蕾:基于Redis的的的分布式锁到底安全吗(上)?

张铁蕾:基于Redis的的的分布式锁到底安全吗(下)?


CSDN文章同步会慢些,欢迎关注微信公众号:挨踢男孩


    




你可能感兴趣的:(分布式)