聊聊flink的Async I/O
// This example implements the asynchronous request and callback with Futures that have the For each #asyncInvoke, an async io operation can be triggered, and once it has been done, {@link ResultFuture} can be passed into callbacks or futures to collect the result data. Callback www.michenggw.com example usage: Future example www.mcyllpt.com usage:
// interface of Java 8's futures (which is the same one followed by Flink's Future)
/**
* An implementation of the 'AsyncFunction' that sends requests and sets the callback.
*/
class AsyncDatabaseRequest extends RichAsyncFunction
/** The database specific client that can issue concurrent requests with callbacks */
private transient DatabaseClient client;
@Override
public void open(Configuration parameters) throws Exception {
client = new DatabaseClient(host, post, credentials);
}
@Override
public void close() throws Exception {
client.close();
}
@Override
public void asyncInvoke(String key, final ResultFuture
// issue the asynchronous request, receive a future for result
final Future
// set the callback to be executed once the request by the client is complete
// the callback simply forwards the result to the result future
CompletableFuture.supplyAsync(new Supplier
@Override
public String get() {
try {
return result.get();
} catch (InterruptedException | ExecutionException e) {
// Normally handled explicitly.
return null;
}
}
}).thenAccept( (String dbResult) -> {
resultFuture.complete(Collections.singleton(new Tuple2<>(key, dbResult)));
});
}
}
// create the original stream
DataStream
// apply the async I/O transformation
DataStream
AsyncDataStream.unorderedWait(stream, new AsyncDatabaseRequest(), 1000, TimeUnit.MILLISECONDS, 100);
本实例展示了flink Async I/O的基本用法，首先是实现AsyncFunction接口，用于编写异步请求逻辑及将结果或异常设置到resultFuture，然后就是使用AsyncDataStream的unorderedWait或orderedWait方法将AsyncFunction作用到DataStream作为transformation；AsyncDataStream的unorderedWait或orderedWait有两个关于async operation的参数，一个是timeout参数用于设置async的超时时间，一个是capacity参数用于指定同一时刻最大允许多少个(并发)async request在执行
AsyncFunction
flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/functions/async/AsyncFunction.java
/**
* A function to trigger Async I/O operation.
*
*
* the result can be collected by calling {@link ResultFuture#complete}. For each async
* operation, its context is stored in the operator immediately after invoking
* #asyncInvoke, avoiding blocking for each stream input as long as the internal buffer is not full.
*
*
* An error can also be propagate to the async IO operator by
* {@link ResultFuture#completeExceptionally(Throwable)}.
*
*
*
* {@code
* public class HBaseAsyncFunc implements AsyncFunction
*
* public void asyncInvoke(String row, ResultFuture
* HBaseCallback cb = new HBaseCallback(result);
* Get get = new Get(Bytes.toBytes(row));
* hbase.asyncGet(get, cb);
* }
* }
* }
*
*
*
* {@code
* public class HBaseAsyncFunc implements AsyncFunction
*
* public void asyncInvoke(String row, final ResultFuture
* Get get = new Get(Bytes.toBytes(row));
* ListenableFuture
* Futures.addCallback(future, new FutureCallback
* public void onSuccess(Result result) {
* List
* result.complete(ret);
* }
* public void onFailure(Throwable thrown) {
* result.completeExceptionally(thrown);
* }
* });
* }
* }
* }
*
* @param
* @param
*/
@PublicEvolving
public interface AsyncFunction
/**
* Trigger async operation for each stream input.
*
* @param input element coming from an upstream task
* @param resultFuture to be completed with the result data
* @exception Exception in case of a user code error. An exception will make the task fail and
* trigger fail-over process.
*/
void asyncInvoke(IN input, ResultFuture
/**
* {@link AsyncFunction#asyncInvoke} timeout occurred.
* By default, the result future is exceptionally completed with a timeout exception.
*
* @param input element coming from an upstream task
* @param resultFuture to be completed with the result data
*/
default void timeout(IN input, ResultFuture
resultFuture.completeExceptionally(
new TimeoutException("Async function call has timed out."));
}
}
AsyncFunction接口继承了Function，它定义了asyncInvoke方法以及一个default的timeout方法；asyncInvoke方法执行异步逻辑，然后通过ResultFuture.complete将结果设置到ResultFuture，如果异常则通过ResultFuture.completeExceptionally(Throwable)来传递到ResultFuture
RichAsyncFunction
flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/functions/async/RichAsyncFunction.java
@PublicEvolving
public abstract class RichAsyncFunction
private static final long serialVersionUID = 3858030061138121840L;
@Override
public void setRuntimeContext(RuntimeContext runtimeContext) {
Preconditions.checkNotNull(runtimeContext);
if (runtimeContext instanceof IterationRuntimeContext) {
super.setRuntimeContext(
new RichAsyncFunctionIterationRuntimeContext(
(IterationRuntimeContext) runtimeContext));
} else {
super.setRuntimeContext(new RichAsyncFunctionRuntimeContext(runtimeContext));
}
}
@Override
public abstract void asyncInvoke(IN input, ResultFuture
//......
}
RichAsyncFunction继承了AbstractRichFunction，同时声明实现AsyncFunction接口，它不没有实现asyncInvoke，交由子类实现；它覆盖了setRuntimeContext方法，这里使用RichAsyncFunctionRuntimeContext或者RichAsyncFunctionIterationRuntimeContext进行包装
RichAsyncFunctionRuntimeContext
flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/functions/async/RichAsyncFunction.java
/**
* A wrapper class for async function's {@link RuntimeContext}. The async function runtime
* context only supports basic operations which are thread safe. Consequently, state access,
* accumulators, broadcast variables and the distributed cache are disabled.
*/
private static class RichAsyncFunctionRuntimeContext implements RuntimeContext {
private final RuntimeContext runtimeContext;
RichAsyncFunctionRuntimeContext(RuntimeContext context) {
runtimeContext = Preconditions.checkNotNull(context);
}
@Override
public String getTaskName() {
return runtimeContext.getTaskName();
}
@Override
public MetricGroup getMetricGroup() {
return runtimeContext.getMetricGroup();
}
@Override
public int getNumberOfParallelSubtasks() {
return runtimeContext.getNumberOfParallelSubtasks();
}
@Override
public int getMaxNumberOfParallelSubtasks() {
return runtimeContext.getMaxNumberOfParallelSubtasks();
}
@Override
public int getIndexOfThisSubtask() {
return runtimeContext.getIndexOfThisSubtask();
}
@Override
public int getAttemptNumber() {
return runtimeContext.getAttemptNumber();
}
@Override
public String getTaskNameWithSubtasks() {
return runtimeContext.getTaskNameWithSubtasks();
}
@Override
public ExecutionConfig getExecutionConfig() {
return runtimeContext.getExecutionConfig();
}
@Override
public ClassLoader getUserCodeClassLoader() {
return runtimeContext.getUserCodeClassLoader();
}
// -----------------------------------------------------------------------------------
// Unsupported operations
// -----------------------------------------------------------------------------------
@Override
public DistributedCache getDistributedCache() {
throw new UnsupportedOperationException("Distributed cache is not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("State is not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("State is not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("State is not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("State is not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("State is not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("State is not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("Accumulators are not supported in rich async functions.");
}
@Override
public
throw new UnsupportedOperationException("Accumulators are not supported in rich async functions.");
}
@Override
public Map
throw new UnsupportedOperationException("Accumulators are not supported in rich async functions.");
}
@Override
public IntCounter getIntCounter(String name) {
throw new UnsupportedOperationException("Int counters are not supported in rich async functions.");
}
@Override
public LongCounter getLongCounter(String name) {
throw new UnsupportedOperationException("Long counters are not supported in rich async functions.");
}
@Override
public DoubleCounter getDoubleCounter(String name) {
throw new UnsupportedOperationException("Long counters are not supported in rich async functions.");
}
@Override
public Histogram getHistogram(String name) {
throw new UnsupportedOperationException("Histograms are not supported in rich async functions.");
}
@Override
public boolean hasBroadcastVariable(String name) {
throw new UnsupportedOperationException("Broadcast variables are not supported in rich async functions.");
}
@Override
public