聊聊PowerJob的FileCleanupProcessor
序
本文主要研究一下PowerJob的FileCleanupProcessor
BroadcastProcessor
tech/powerjob/worker/core/processor/sdk/BroadcastProcessor.java
public interface BroadcastProcessor extends BasicProcessor {
/**
* 在所有节点广播执行前执行，只会在一台机器执行一次
*/
default ProcessResult preProcess(TaskContext context) throws Exception {
return new ProcessResult(true);
}
/**
* 在所有节点广播执行完成后执行，只会在一台机器执行一次
*/
default ProcessResult postProcess(TaskContext context, List taskResults) throws Exception {
return defaultResult(taskResults);
}
static ProcessResult defaultResult(List taskResults) {
long succeed = 0, failed = 0;
for (TaskResult ts : taskResults) {
if (ts.isSuccess()) {
succeed ++ ;
}else {
failed ++;
}
}
return new ProcessResult(failed == 0, String.format("succeed:%d, failed:%d", succeed, failed));
}
}
BroadcastProcessor继承了BasicProcessor，它提供了preProcess、postProcess方法
FileCleanupProcessor
tech/powerjob/official/processors/impl/FileCleanupProcessor.java
public class FileCleanupProcessor implements BroadcastProcessor {
@Override
public ProcessResult preProcess(TaskContext context) throws Exception {
if (SecurityUtils.disable(SecurityUtils.ENABLE_FILE_CLEANUP_PROCESSOR)) {
String msg = String.format("FileCleanupProcessor is not enabled, please set '-D%s=true' to enable it", SecurityUtils.ENABLE_FILE_CLEANUP_PROCESSOR);
context.getOmsLogger().warn(msg);
return new ProcessResult(false, msg);
}
return new ProcessResult(true);
}
@Override
public ProcessResult process(TaskContext taskContext) throws Exception {
OmsLogger logger = taskContext.getOmsLogger();
logger.info("using params: {}", taskContext.getJobParams());
LongAdder cleanNum = new LongAdder();
Stopwatch sw = Stopwatch.createStarted();
List cleanupParamsList = JSONArray.parseArray(taskContext.getJobParams(), CleanupParams.class);
cleanupParamsList.forEach(params -> {
logger.info("start to process: {}", JSON.toJSON(params));
if (StringUtils.isEmpty(params.filePattern) || StringUtils.isEmpty(params.dirPath)) {
logger.warn("skip due to invalid params!");
return;
}
File dir = new File(params.dirPath);
if (!dir.exists()) {
logger.warn("skip due to dirPath[{}] not exists", params.dirPath);
return;
}
if (!dir.isDirectory()) {
logger.warn("skip due to dirPath[{}] is not a directory", params.dirPath);
return;
}
logger.info("start to search directory: {}", params.dirPath);
Collection files = FileUtils.listFiles(dir, null, true);
logger.info("total file num: {}", files.size());
Pattern filePattern = Pattern.compile(params.filePattern);
files.forEach(file -> {
String fileName = file.getName();
String filePath = file.getAbsolutePath();
if (!filePattern.matcher(fileName).matches()) {
logger.info("file[{}] won't be deleted due to filename not match the pattern: {}", fileName, params.filePattern);
return;
}
// last modify time interval, xxx hours
int interval = (int) Math.ceil((System.currentTimeMillis() - file.lastModified()) / 3600000.0);
if (interval < params.retentionTime) {
logger.info("file[{}] won't be deleted because it does not meet the time requirement", filePath);
return;
}
try {
FileUtils.forceDelete(file);
cleanNum.increment();
logger.info("delete file[{}] successfully!", filePath);
} catch (Exception e) {
logger.error("delete file[{}] failed!", filePath, e);
}
});
});
return new ProcessResult(true, String.format("cost:%s,clean:%d", sw.toString(), cleanNum.longValue()));
}
@Data
public static class CleanupParams {
private String dirPath;
private String filePattern;
private Integer retentionTime;
}
}
FileCleanupProcessor实现了BroadcastProcessor接口，其preProcess判断是否开启
powerjob.official-processor.file-cleanup.enable，其process方法接收List参数，然后遍历指定目录，找到匹配的文件，判断当前时间距离最后修改时间是否大于等于retentionTime，是则执行FileUtils.forceDelete(file)
handleBroadcastRootTask
tech/powerjob/worker/core/processor/runnable/HeavyProcessorRunnable.java
private void handleBroadcastRootTask(Long instanceId, TaskContext taskContext) {
BasicProcessor processor = processorBean.getProcessor();
ProcessResult processResult;
// 广播执行的第一个 task 只执行 preProcess 部分
if (processor instanceof BroadcastProcessor) {
BroadcastProcessor broadcastProcessor = (BroadcastProcessor) processor;
try {
processResult = broadcastProcessor.preProcess(taskContext);
} catch (Throwable e) {
log.warn("[ProcessorRunnable-{}] broadcast task preProcess failed.", instanceId, e);
processResult = new ProcessResult(false, e.toString());
}
} else {
processResult = new ProcessResult(true, "NO_PREPOST_TASK");
}
// 通知 TaskTracker 创建广播子任务
reportStatus(processResult.isSuccess() ? TaskStatus.WORKER_PROCESS_SUCCESS : TaskStatus.WORKER_PROCESS_FAILED, suit(processResult.getMsg()), ProcessorReportTaskStatusReq.BROADCAST, taskContext.getWorkflowContext().getAppendedContextData());
}
HeavyProcessorRunnable的handleBroadcastRootTask方法针对BroadcastProcessor先执行preProcess方法
handleLastTask
private void handleLastTask(String taskId, Long instanceId, TaskContext taskContext, ExecuteType executeType) {
final BasicProcessor processor = processorBean.getProcessor();
ProcessResult processResult;
Stopwatch stopwatch = Stopwatch.createStarted();
log.debug("[ProcessorRunnable-{}] the last task(taskId={}) start to process.", instanceId, taskId);
List taskResults = workerRuntime.getTaskPersistenceService().getAllTaskResult(instanceId, task.getSubInstanceId());
try {
switch (executeType) {
case BROADCAST:
if (processor instanceof BroadcastProcessor) {
BroadcastProcessor broadcastProcessor = (BroadcastProcessor) processor;
processResult = broadcastProcessor.postProcess(taskContext, taskResults);
} else {
processResult = BroadcastProcessor.defaultResult(taskResults);
}
break;
case MAP_REDUCE:
if (processor instanceof MapReduceProcessor) {
MapReduceProcessor mapReduceProcessor = (MapReduceProcessor) processor;
processResult = mapReduceProcessor.reduce(taskContext, taskResults);
} else {
processResult = new ProcessResult(false, "not implement the MapReduceProcessor");
}
break;
default:
processResult = new ProcessResult(false, "IMPOSSIBLE OR BUG");
}
} catch (Throwable e) {
processResult = new ProcessResult(false, e.toString());
log.warn("[ProcessorRunnable-{}] execute last task(taskId={}) failed.", instanceId, taskId, e);
}
TaskStatus status = processResult.isSuccess() ? TaskStatus.WORKER_PROCESS_SUCCESS : TaskStatus.WORKER_PROCESS_FAILED;
reportStatus(status, suit(processResult.getMsg()), null, taskContext.getWorkflowContext().getAppendedContextData());
log.info("[ProcessorRunnable-{}] the last task execute successfully, using time: {}", instanceId, stopwatch);
}
HeavyProcessorRunnable的handleLastTask方法针对BroadcastProcessor类型的会执行其postProcess方法
小结
PowerJob的FileCleanupProcessor实现了BroadcastProcessor接口，其preProcess判断是否开启powerjob.official-processor.file-cleanup.enable，其process方法接收List参数，然后遍历指定目录，找到匹配的文件，判断当前时间距离最后修改时间是否大于等于retentionTime，是则执行FileUtils.forceDelete(file)。