ES 客户端BulkProcessor代码解析-代码设计是什么？

ES 客户端BulkProcessor代码解析-代码设计是什么？

BulkProcessor 是ES客户端提供的ES请求处理器， 表现的是一种异步（非阻塞）、批量、有通知回调的请求处理器模式示例。让我们学习如何构建一段优雅的代码

构造器模式构建实例

构建一个Builder类， 设置参数返回builder对象本身，可以持续的调用设置参数方法。并提供合适的默认值减少设置。

BulkProcessor.Builder的参数设置方法如下：

public BulkProcessor.Builder setConcurrentRequests(int concurrentRequests) {     this.concurrentRequests = concurrentRequests;     return this; }

private Builder(BiConsumer> consumer, BulkProcessor.Listener listener, Scheduler scheduler, Runnable onClose) {     this.concurrentRequests = 1;     this.bulkActions = 1000;     this.bulkSize = new ByteSizeValue(5L, ByteSizeUnit.MB);     this.flushInterval = null;     this.backoffPolicy = BackoffPolicy.exponentialBackoff();     this.consumer = consumer;     this.listener = listener;     this.scheduler = scheduler;     this.onClose = onClose; }

Builder的使用代码示例：

return BulkProcessor.builder(bulkConsumer, new BulkProcessor.Listener() {     public void beforeBulk(long executionId, BulkRequest request) {     }     public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {     }     public void afterBulk(long executionId, BulkRequest request, Throwable failure) {     } }).setBulkActions(this.get_bulkActions()).setBulkSize(new ByteSizeValue(this.get_bulkSize(), ByteSizeUnit.MB)).setFlushInterval(TimeValue.timeValueSeconds(this.get_flushInterval())).setConcurrentRequests(this.get_concurrentRequests()).build();

区别核心及变化点、开放变换点提供扩展性

充分的扩展

这一点是优秀设计最重要的判断条件， 保证功能健壮性和可扩展性的设计方法。

要处理任务任务，通常来说会设计一个接受任务的方法， 同时在任务处理模块中建立列表等各种数据结构来保存。BuikProcessor类设计上将请求的执行和请求的管理区别开，核心实现请求的执行，而将请求的管理（产生）通过函数式编程的Supplier接口交由调用者负责提供。

private final Supplier bulkRequestSupplier;

private void execute() {     BulkRequest bulkRequest = this.bulkRequest;     long executionId = this.executionIdGen.incrementAndGet();     this.bulkRequest = (BulkRequest)this.bulkRequestSupplier.get();     this.bulkRequestHandler.execute(bulkRequest, executionId); }

BiConsumer> bulkConsumer 提供实际的请求执行扩展。

BiConsumer> bulkConsumer = (request, bulkListener) -> {     RestClientHelper.getClient().bulkAsync(request, RequestOptions.DEFAULT, bulkListener); };

BulkProcessor.Listener： 提供执行回调的扩展。

控制反转

       通过向使用者要求产生请求的功能而不是直接接受请求，将请求控制由无法掌控的外界转为自己张控制（推送转为拉）

       通过定时任务定时拉取处理。

private Cancellable startFlushTask(TimeValue flushInterval, Scheduler scheduler) {     if (flushInterval == null) {         return new Cancellable() {             public boolean cancel() {                 return false;             }             public boolean isCancelled() {                 return true;             }         };     } else {         Runnable flushRunnable = scheduler.preserveContext(new BulkProcessor.Flush());         return scheduler.scheduleWithFixedDelay(flushRunnable, flushInterval, "generic");     } }

class Flush implements Runnable {     Flush() {     }     public void run() {         synchronized(BulkProcessor.this) {             if (!BulkProcessor.this.closed) {                 if (BulkProcessor.this.bulkRequest.numberOfActions() != 0) {                     BulkProcessor.this.execute();                 }             }         }     } }

兼容传统

在进行创新的拉取控制，提供传统直接添加请求的方法

public BulkProcessor add(IndexRequest request) {     return this.add((DocWriteRequest)request); } public BulkProcessor add(DeleteRequest request) {     return this.add((DocWriteRequest)request); }

构建一个异步的处理 内核—抽象设计

调度器

调度器Scheduler： 负责构建底层执行引擎（线程池）；实现可取消、延迟处理、

线程池

static ScheduledThreadPoolExecutor initScheduler(Settings settings) {     ScheduledThreadPoolExecutor scheduler = new Scheduler.SafeScheduledThreadPoolExecutor(1, EsExecutors.daemonThreadFactory(settings, "scheduler"), new EsAbortPolicy());     scheduler.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);     scheduler.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);     scheduler.setRemoveOnCancelPolicy(true);     return scheduler; }

任务可延迟、可取消。 提供帮助类将普通任务封装为可取消、可延迟的任务。

class ScheduledCancellableAdapter implements ScheduledCancellable {     private final ScheduledFuture scheduledFuture;     ScheduledCancellableAdapter(ScheduledFuture scheduledFuture) {         assert scheduledFuture != null;         this.scheduledFuture = scheduledFuture;     }     public long getDelay(TimeUnit unit) {         return this.scheduledFuture.getDelay(unit);     }     public int compareTo(Delayed other) {         return -other.compareTo(this.scheduledFuture);     }     public boolean cancel() {         return FutureUtils.cancel(this.scheduledFuture);     }     public boolean isCancelled() {         return this.scheduledFuture.isCancelled();     } }

static Scheduler.ScheduledCancellable wrapAsScheduledCancellable(ScheduledFuture scheduledFuture) {     return new ScheduledCancellableAdapter(scheduledFuture); }

任务处理器

       BulkRequestHandler 负责处理具体任务:负责任务的执行、重试、调用实际的请求处理器、调用执行回调等。

       执行入口（BulkProcessor调用）：

private void execute() {     BulkRequest bulkRequest = this.bulkRequest;     long executionId = this.executionIdGen.incrementAndGet();     this.bulkRequest = (BulkRequest)this.bulkRequestSupplier.get();     this.bulkRequestHandler.execute(bulkRequest, executionId); }

       BulkRequestHandler是一个线程安全类。执行代码解析：

public void execute(final BulkRequest bulkRequest, final long executionId) {   Runnable toRelease = () -> {   };   boolean bulkRequestSetupSuccessful = false;   try {       //调用监听器的前置处理       this.listener.beforeBulk(executionId, bulkRequest);       //信号量并发控制       this.semaphore.acquire();       Semaphore var10000 = this.semaphore;       Objects.requireNonNull(var10000);       toRelease = var10000::release;       //CountDownLatch并发通知       final CountDownLatch latch = new CountDownLatch(1);       //retry异步执行，执行后进行信号量释放、CountDownLatch释放通知       this.retry.withBackoff(this.consumer, bulkRequest, new ActionListener() {           public void onResponse(BulkResponse response) {               try {                   BulkRequestHandler.this.listener.afterBulk(executionId, bulkRequest, response);               } finally {                   BulkRequestHandler.this.semaphore.release();                   latch.countDown();               }           }           public void onFailure(Exception e) {               try {                   BulkRequestHandler.this.listener.afterBulk(executionId, bulkRequest, e);               } finally {                   BulkRequestHandler.this.semaphore.release();                   latch.countDown();               }           }       });       bulkRequestSetupSuccessful = true;       //当并发量达上线，等待释放       if (this.concurrentRequests == 0) {           latch.await();       }   } catch (InterruptedException var11) {       Thread.currentThread().interrupt();       this.logger.info(() -> {           return new ParameterizedMessage("Bulk request {} has been cancelled.", executionId);       }, var11);       this.listener.afterBulk(executionId, bulkRequest, var11);   } catch (Exception var12) {       this.logger.warn(() -> {           return new ParameterizedMessage("Failed to execute bulk request {}.", executionId);       }, var12);       //异常回调       this.listener.afterBulk(executionId, bulkRequest, var12);   } finally {       if (!bulkRequestSetupSuccessful) {           toRelease.run();       }   } }

重试策略：BackoffPolicy，

通过Iteratble接口的抽象，以hasNext 和 Next来确定能否重试及重试的延迟时间。

public abstract class BackoffPolicy implements Iterable

ConstantBackoff: 固定频率+固定时间的重试

NoBackOff: 无重试

ExponentialBackOff: 指数型重试。

private static class ExponentialBackoffIterator implements Iterator {     private final int numberOfElements;     private final int start;     private int currentlyConsumed;     private ExponentialBackoffIterator(int start, int numberOfElements) {         this.start = start;         this.numberOfElements = numberOfElements;     }     public boolean hasNext() {         return this.currentlyConsumed < this.numberOfElements;     }     public TimeValue next() {         if (!this.hasNext()) {             throw new NoSuchElementException("Only up to " + this.numberOfElements + " elements");         } else {             int result = this.start + 10 * ((int)Math.exp(0.8D * (double)this.currentlyConsumed) - 1);             ++this.currentlyConsumed;             return TimeValue.timeValueMillis((long)result);         }     } }

异步执行抽象：Retry（使用）

private void retry(BulkRequest bulkRequestForRetry) {     assert this.backoff.hasNext();     TimeValue next = (TimeValue)this.backoff.next();     logger.trace("Retry of bulk request scheduled in {} ms.", next.millis());     Runnable command = this.scheduler.preserveContext(() -> {         this.execute(bulkRequestForRetry);     });     this.retryCancellable = this.scheduler.schedule(command, next, "same"); }

RetryHandler implements ActionListener 作为执行器入口，同时充当实际执行器的回调Listener的角色。

public void execute(BulkRequest bulkRequest) {     this.currentBulkRequest = bulkRequest;     this.consumer.accept(bulkRequest, this); }

public void onResponse(BulkResponse bulkItemResponses) {     if (!bulkItemResponses.hasFailures()) {         this.addResponses(bulkItemResponses, (r) -> {             return true;         });         this.finishHim();     } else if (this.canRetry(bulkItemResponses)) {         this.addResponses(bulkItemResponses, (r) -> {             return !r.isFailed();         });         this.retry(this.createBulkRequestForRetry(bulkItemResponses));     } else {         this.addResponses(bulkItemResponses, (r) -> {             return true;         });         this.finishHim();     } }

其他包括onFailure、finishHim等方法负责执行对应的回调行为（同时业务回调接口）。

      

任务批处理

批处理包括请求缓存、批量处理。

this.bulkRequest.add(request); this.executeIfNeeded();

private void executeIfNeeded() {     this.ensureOpen();     if (this.isOverTheLimit()) {         this.execute();     } }

private boolean isOverTheLimit() {     if (this.bulkActions != -1 && this.bulkRequest.numberOfActions() >= this.bulkActions) {         return true;     } else {         return this.bulkSize != -1L && this.bulkRequest.estimatedSizeInBytes() >= this.bulkSize;     } }

请求需要具备唯一标识

private final AtomicLong executionIdGen = new AtomicLong();

long executionId = this.executionIdGen.incrementAndGet();

优雅的关闭

public synchronized boolean awaitClose(long timeout, TimeUnit unit) throws InterruptedException {     if (this.closed) {         return true;     } else {         this.closed = true;         this.cancellableFlushTask.cancel();         if (this.bulkRequest.numberOfActions() > 0) {             this.execute();         }         boolean var4;         try {             var4 = this.bulkRequestHandler.awaitClose(timeout, unit);         } finally {             this.onClose.run();         }         return var4;     } }

线程安全

请求添加方法需要考虑安全性， 这里使用synchronized

public synchronized BulkProcessor add

总结

       优秀的代码设计是的产品高效、健壮和强扩展性。BulkProcessor ES请求处理器的时间体现了以下优秀的设计实践：

1. 核心和变动点分开
2. 简洁而不简单的使用方式， 大家可以思考一下设计流行的做减法代表着什么？
3. 深入灵活的结构抽象。