springBoot框架下利用Apache Camel 实现自动同步采集 FTP服务器上文件数据

一、需要引用apache.camel 相关包。

build.gradle 定义如下:

buildscript {
    ext {
        springBootVersion = '2.1.1.RELEASE'
    }
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
    }
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'

group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = 1.8

repositories {
    mavenCentral()
}


dependencies {
    implementation('org.springframework.boot:spring-boot-starter-web')

    implementation('org.apache.camel:camel-spring-boot-starter:2.23.0')
    implementation('org.apache.camel:camel-core:2.23.0')
    implementation('org.apache.camel:camel-ftp:2.23.0')

    testImplementation('org.springframework.boot:spring-boot-starter-test')
}

二、编写一个Component，扩展RouteBuilder.class,重新实现configure方法。

设置数据源、目标，并且可以根据需要额外设置 过滤器Predicate和处理器Processor。

package com.example.ftpdemo;

import org.apache.camel.LoggingLevel;
import org.apache.camel.builder.RouteBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

/**
 * @className: DownloadRouteDemo
 * @description: DownloadRouteDemo
 * @author: houwei
 * @date: 2018/12/26 14:23
 */
@Component
public class DownloadRouteDemo extends RouteBuilder {

    private static Logger logger = LoggerFactory.getLogger(DownloadRouteDemo.class);

    @Value("${ftp.server.info}")
    private String ftpServer;
    @Value("${ftp.local.dir}")
    private String downloadLocation;

    @Autowired
    LocationFileProcessor locationFileProcessor;
    @Autowired
    FtpDownloadFileFilter ftpDownloadFileFilter;

    @Override
    public void configure() throws Exception {
        from(ftpServer)    //数据源
                .filter(ftpDownloadFileFilter) //过滤器
                .to(downloadLocation) //目标
                .log(LoggingLevel.INFO, logger, "Downloaded file ${file:name} complete.")
                .process(locationFileProcessor); //处理器
    }
}

过滤器设置，只处理.txt或者.json文件。上述代码中 filter 必须再to 前面才能生效。

package com.example.ftpdemo;

import org.apache.camel.Exchange;
import org.apache.camel.Predicate;
import org.apache.camel.component.file.GenericFileMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

import java.io.RandomAccessFile;


/**
 * @className: FtpDownloadFileFilter
 * @description: FtpDownloadFileFilter
 * @author: houwei
 * @date: 2018/12/26 16:37
 */

@Component
public class FtpDownloadFileFilter implements Predicate {
    private static Logger logger = LoggerFactory.getLogger( FtpDownloadFileFilter.class );

    @Override
    public boolean matches(Exchange exchange) {
        boolean bMatches;
        GenericFileMessage inFileMessage = (GenericFileMessage) exchange.getIn();
        String fileName = inFileMessage.getGenericFile().getFileName();
        bMatches = fileName.endsWith(".txt") || fileName.endsWith(".json");

        logger.info("fileName:"+fileName+"matches:"+bMatches);

        return bMatches;
    }
}

处理器代码如下:

package com.example.ftpdemo;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.component.file.GenericFileMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

import java.io.RandomAccessFile;

/**
 * @className: LocationFileProcessor
 * @description: LocationFileProcessor
 * @author: houwei
 * @date: 2018/12/26 14:24
 */
@Component
public class LocationFileProcessor implements Processor {

    private static Logger logger = LoggerFactory.getLogger( LocationFileProcessor.class );


    @Value("${ftp.local.dir}")
    private String fileDir;

    @Autowired
    OrderService orderService;

    @Override
    public void process(Exchange exchange) throws Exception {
        GenericFileMessage inFileMessage = (GenericFileMessage) exchange.getIn();
        //文件名
        String fileName = inFileMessage.getGenericFile().getFileName();
        //文件的绝对路径
        logger.info(fileDir + fileName);
        //解析入库等操作
        orderService.process(fileDir + fileName);

    }

}

定义一个OrderService

package com.example.ftpdemo;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;

/**
 * @className: OrderService
 * @description: OrderService
 * @author: houwei
 * @date: 2018/12/26 14:26
 */
@Service
public class OrderService {
    private static Logger logger = LoggerFactory.getLogger( OrderService.class );
    public void process(String path){
        logger.info("path:"+path);
    }
}

三、application.properties 定义如下:

camel.springboot.main-run-controller=true

management.endpoint.camelroutes.enabled=true
management.endpoint.camelroutes.read-only=true

ftp.server.info=ftp://[email protected]:21/rtdata?password=123456&delay=5s&readLock=rename&move=done

ftp.local.dir=file:D:/temp/ftp/

参数说明：
ftp://[email protected]:21/rtdata?password=123456&delay=5s&readLock=rename&move=done 

delay 采样时间频率

move 表示下载文件完成后，ftp端文件移动到当前目录下的done目录下

详细参数参考：http://camel.apache.org/ftp2.html