springBatch的决策器和监听器
https://blog.csdn.net/sdsxdqg/article/details/108742331
读https://www.freesion.com/article/4817911665/
数据输出:
ItemReader是一个数据一个数据的读,而ItemWriter是一批一批的输出
数据输出到普通文件;flatFileItemWriter
SpringBatch是一个轻量级的、完善的批处理框架,旨在帮助企业建立健壮、高效的批处理应用。SpringBatch是Spring的一个子项目,使用Java语言并基于Spring框架,使得已经使用Spring的开发者和企业更容易访问和利用企业服务。
Spring Batch提供了大量可重用的组件,包括了日志、追踪、事务、任务作业统计、任务重启、跳过、重复、资源管理。对于大数据和高性能的批处理任务,Spring Batch同样提供了高级功能来支持,如分区功能、远程功能。Spring Batch能够支持简单的、复杂的和大数据量的批处理作业。
SpringBatch是一个批处理应用框架,不是一个调度框架。需要和调度框架合作来构建完成批处理任务。它只关注批处理任务相关的问题。如事务、并发、监控、执行等。并不提供相应的调度功能。
@Configuration
@EnableBatchProcessing
public class JobConfiguration {
//注入创建任务对象的对象
@Autowired
private JobBuilderFactory jobBuilderFactory;
//任务的执行由step决定
//注入创建Step对象的对象
@Autowired
private StepBuilderFactory stepBuilderFactory;
//创建任务对象
@Bean
public Job helloWorldJob(){
return jobBuilderFactory.get("helloWorldJob")
.start(step1())
.build();
}
@Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("hello world!");
return RepeatStatus.FINISHED; //正常结束
}
}).build();
}
}
创建Step有两种方式一种时tasklet一种时chunk
当由tasklet创建时只有返回RepeatStatus中一个正常的值,才会正常的执行下去,才会去执行下一个任务。
SpringBatch在执行任务的过程中,会把一些数据持久化数据库中,在本次学习SpringBatch的过程中我这里主要用的时MySQL数据库。
<dependency>
<groupId>mysqlgroupId>
<artifactId>mysql-connector-javaartifactId>
<version>5.1.47version>
dependency>
<dependency>
<groupId>org.springframework.bootgroupId>
<artifactId>spring-boot-starter-jdbcartifactId>
dependency>
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://ip:3306/springbatch
spring.datasource.username=springbatch
spring.datasource.password=springbatch
spring.datasource.schema=classpath:/org/springframework/batch/core/schema-mysql.sql # 会创建对应的表
spring.batch.initialize-schema=always
spring.batch.initialize-schema=always在初始化的时候会创建spring.datasource.schema=classpath:/org/springframework/batch/core/schema-mysql.sql 中对应的表,有了表我们才能存储持久化的数据
JobInstance
Job在执行时会执行一个instance,每一个instance都会对应一个job_execution会记录job执行的成功与否。
在执行Job时如果失败,在下一次执行时这个job对应的还是这个instance但是会有一个新的job_execution
JobParameters:
是一组可以贯穿整个Job的运行时配置参数,不同的配置将产生不同的Instance,如果你是使用相同的JobParameters运行同一个Job,那么运行的Job将会重用上一个的Instance
JobExecution
该领域概念表示JobInstance的一次运行,JobInstance运行时可能成功或者失败,每一次JobInstance的运行都会产生一个JobExecution,
StepExecution
类似于JobExecution,该领域对象表示Step的一次运行,Step是Job的一部分,因此一个StepExecution会关联到一个Jobexecution,令外,改对象还会存储很多与这次Step运行相关的所有数据,因此该对象也就有很多的属性,并且需要持久化以支持SpringBatch的特性
ExecutionContext:
从前面的JobExecution,StepExecution的属性介绍中已经提到了该领域的概念,该领域就是一个容器,该容器由Batch 框架控制,框架会对该容器持久化,开发人员可以使用该容器保存一些数据,以支持在整个batchJob或者整个Step中共享这些数据
Job:作业,批处理中的核心概念,是Batch操作的基础单元。每个作业Job有1个或者多个作业步Step。
step的执行有两种方式,如下
//@Configuration
//@EnableBatchProcessing
public class JobDemo {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
//执行三个Step,最后调用build()方法,
// 第一种按照star() next()的方式执行step按顺序执行
// @Bean
// public Job jobDemoJob(){
// return jobBuilderFactory.get("jobDemoJob")
// .start(step1())
// .next(step2())
// .next(step3())
// .build();
// }
//on 是用来指定条件的,从哪里到哪里
@Bean
public Job jobDemoJob(){
return jobBuilderFactory.get("jobDemoJob")
.start(step1())
.on("COMPLETED").to(step2())
.from(step2()).on("COMPLETED").to(step3()) //其他的一些方法 fail() stopAndRestart
.from(step3()).end()
.build();
}
@Bean
public Step step3() {
return stepBuilderFactory.get("step3")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("step3");
return RepeatStatus.FINISHED;
}
}).build();
}
@Bean
public Step step2() {
return stepBuilderFactory.get("step2")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("step2");
return RepeatStatus.FINISHED;
}
}).build();
}
@Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("step1");
return RepeatStatus.FINISHED;
}
}).build();
}
}
1、Flow 是多个Step的集合
2、可以被多个Job复用
3、使用FlowBuilder来创建
flow和step可以结合着使用
@Configuration
@EnableBatchProcessing
public class FlowDemo {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Bean
public Step flowDemoStep1(){
return stepBuilderFactory.get("flowDemoStep1")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("flowDemoStep1");
return RepeatStatus.FINISHED;
}
}).build();
}
@Bean
public Step flowDemoStep2(){
return stepBuilderFactory.get("flowDemoStep2")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("flowDemoStep2");
return RepeatStatus.FINISHED;
}
}).build();
}
@Bean
public Step flowDemoStep3(){
return stepBuilderFactory.get("flowDemoStep3")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("flowDemoStep3");
return RepeatStatus.FINISHED;
}
}).build();
}
//创建Flow对象,指明Flow对象包含那些step
@Bean
public Flow flowDemoFlow(){
return new FlowBuilder<Flow>("flowDemoFlow")
.start(flowDemoStep1()) //指定flow中包含那些step
.next(flowDemoStep2())
.build();
}
//创建Job对象
//flow和step可以结合着使用
@Bean
public Job flowDemoJob(){
return jobBuilderFactory.get("flowDemoJob")
.start(flowDemoFlow())
.next(flowDemoStep3())
.end().build();
}
}
实现任务中的多个step或者多个flow并发执行
1、创建若干个step
2、创建两个flow
3、创建一个任务包含两个以上flow,并让这两个flow并发执行
.split(new SimpleAsyncTaskExecutor()).add(splitDemoFlow2())
@Configuration
@EnableBatchProcessing
public class SplitDemo {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Bean
public Step splitDEmoStep0(){
return stepBuilderFactory.get("splitDEmoStep0")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("splitDEmoStep0");
return RepeatStatus.FINISHED;
}
}).build();
}
@Bean
public Step splitDEmoStep1(){
return stepBuilderFactory.get("splitDEmoStep1")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("splitDEmoStep1");
return RepeatStatus.FINISHED;
}
}).build();
}
@Bean
public Step splitDEmoStep2(){
return stepBuilderFactory.get("splitDEmoStep2")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("splitDEmoStep2");
return RepeatStatus.FINISHED;
}
}).build();
}
@Bean
public Step splitDEmoStep3(){
return stepBuilderFactory.get("splitDEmoStep3")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
System.out.println("splitDEmoStep3");
return RepeatStatus.FINISHED;
}
}).build();
}
//创建flow
@Bean
public Flow splitDemoFlow1(){
return new FlowBuilder<Flow>("splitDemoFlow1")
.start(splitDEmoStep0())
.next(splitDEmoStep1())
.build();
}
@Bean
public Flow splitDemoFlow2(){
return new FlowBuilder<Flow>("splitDemoFlow2")
.start(splitDEmoStep2())
.next(splitDEmoStep3())
.build();
}
//创建任务
@Bean
public Job splitDemoJob(){
return jobBuilderFactory.get("splitDemoJob")
.start(splitDemoFlow1())
.split(new SimpleAsyncTaskExecutor()).add(splitDemoFlow2())
.end().build();
}
}
ItemProcessor用于处理业务逻辑,验证、过滤等功能,
在本节的案例中,是从从文件中读取,然后打印输出,工程名为:itemprocessor
当有多个ItemProcessor时,用CompositeItemProcessor处理。
关键代码如下:
@Bean
public Step itemProcessorDemo12Step() {
return stepBuilderFactory.get("itemProcessorDemo12Step")
.<Customer,Customer>chunk(3)
.reader(flatFileItemReader())
// .processor(nameUpperProcessor) //只使用一个itemProcessor时
.processor(process()) //使用多种itemProcessor时
.writer(flatFileItemWriter)
.build();
}
//有多种处理数据的方式
@Bean
public CompositeItemProcessor<Customer,Customer> process(){
CompositeItemProcessor<Customer, Customer> processor = new CompositeItemProcessor<>();
List<ItemProcessor<Customer,Customer>> delegates = new ArrayList<>();
delegates.add(nameUpperProcessor);
delegates.add(idFilterProcessor);
processor.setDelegates(delegates);
return processor;
}
@Component
public class IdFilterProcessor implements ItemProcessor<Customer,Customer> {
@Override
public Customer process(Customer customer) throws Exception {
if (customer.getId()%2==0){
return customer;
}
return null;
}
}
接口 :JobExecutionDecider
我们自定义的决策器
/**
* 决策器对应的实现类
*/
public class MyDecider implements JobExecutionDecider {
private int count;
@Override
public FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
//每次执行count+1
count ++;
if ((count%2 ==0)){
//偶数时返回even
return new FlowExecutionStatus("even");
}
//奇数时返回odd
return new FlowExecutionStatus("odd");
}
}
//@Configuration
//@EnableBatchProcessing
public class DeciderDemo {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Bean
public Job DeciderDemoJob(){
return jobBuilderFactory.get("DeciderDemoJob")
.start(deciderDemoStep1())
.next(myDecider())
.from(myDecider()).on("even").to(deciderDemoStep2()) //如果决策器返回的是偶数,则让job执行step2
.from(myDecider()).on("odd").to(deciderDemoStep3())//如果决策器返回的是基数,则让job执行step3
.from(deciderDemoStep3()).on("*").to(myDecider())//执行完step3无论返回什么都返回到我们的决策器
.end().build();
}
@Bean
public Step deciderDemoStep1(){
return stepBuilderFactory.get("deciderDemoStep1")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
System.out.println("deciderDemoStep1");
return RepeatStatus.FINISHED;
}
})
.build();
}
@Bean
public Step deciderDemoStep2(){
return stepBuilderFactory.get("deciderDemoStep2")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
System.out.println("deciderDemoStep2");
return RepeatStatus.FINISHED;
}
})
.build();
}
@Bean
public Step deciderDemoStep3(){
return stepBuilderFactory.get("deciderDemoStep3")
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
System.out.println("deciderDemoStep3");
return RepeatStatus.FINISHED;
}
})
.build();
}
//创建决策器
@Bean
public JobExecutionDecider myDecider(){
return new MyDecider();
}
}
一个job可以嵌套在另一个Job中,被嵌套的Job被称为子Job,外部Job被称为父Job,子Job不能单独执行,需要父Job来启动。
案例:创建两个Job,一个作为子Job,一个作为父Job
在properties文件中指定启动那个job
#加入启动的Job的名称
spring.batch.job.names=parentJob
@Configuration
public class NestedDemo {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Autowired
private Job childJobOne;
@Autowired
private Job childJobTwo;
@Autowired
private JobLauncher jobLauncher;
@Bean
public Job parentJob(JobRepository jobRepository, PlatformTransactionManager transactionManager){
return jobBuilderFactory.get("parentJob")
.start(childJob11(jobRepository,transactionManager))
.next(childJob21(jobRepository,transactionManager))
.build();
}
//返回的是Job类型的step,特殊的Step
@Bean
public Step childJob21(JobRepository jobRepository, PlatformTransactionManager transactionManager) {
return new JobStepBuilder(new StepBuilder("childJob21"))
.job(childJobTwo)
.launcher(jobLauncher) //使用启动父Job的启动对象
.repository(jobRepository) //持久化
.transactionManager(transactionManager) //事务管理
.build();
}
@Bean
public Step childJob11(JobRepository jobRepository, PlatformTransactionManager transactionManager) {
return new JobStepBuilder(new StepBuilder("childJob11"))
.job(childJobOne)
.launcher(jobLauncher) //使用启动父Job的启动对象
.repository(jobRepository) //持久化
.transactionManager(transactionManager) //事务管理
.build();
}
}
用来监听批处理作业的执行情况,创建监听可以通过接口或者使用注解
1、可以监听任务
2、可以监听step
/**
* 监听任务的监听器 Job级别的 接口形式
* @author tqq
* @date 2021年08月23日 22:17
*/
public class MyJobListener implements JobExecutionListener {
@Override
public void beforeJob(JobExecution jobExecution) {
System.out.println("===任务"+jobExecution.getJobInstance().getJobName()+"执行前===");
}
@Override
public void afterJob(JobExecution jobExecution) {
System.out.println("===任务"+jobExecution.getJobInstance().getJobName()+"执行后===");
}
/**
* 监听任务的监听器 注解形式
* @author tqq
* @date 2021年08月23日 22:21
*/
public class MyChunkListener {
@BeforeChunk
public void beforeChunk(ChunkContext context){//chunk的上下文
System.out.println(context.getStepContext().getStepName()+"before.....");
}
@AfterChunk
public void afterChunk(ChunkContext context){
System.out.println(context.getStepContext().getStepName()+"after.....");
}
}
@Configuration
@EnableBatchProcessing
public class ListenerDemo {
//注入创建任务对象的对象
@Autowired
private JobBuilderFactory jobBuilderFactory;
//注入创建step对象的对象
@Autowired
private StepBuilderFactory stepBuilderFactory;
//创建任务对象
@Bean
public Job jobListenerJob1(){
return jobBuilderFactory.get("jobListenerJob1")
.start(step1())
.listener(getJobListener())
.build();
}
@Bean
public Step step1(){
return stepBuilderFactory.get("step1")
.<String,String>chunk(2)//每次读取两个数据 read process writer
.faultTolerant()//容错的
.listener(new MyChunkListener())
.reader(read())
.writer(writer())
.build();
}
@Bean
public ItemWriter<String> writer() {
return new ItemWriter<String>() {
@Override
public void write(List<? extends String> list) throws Exception {
for (String s : list) {
System.out.println(s);
}
}
};
}
@Bean
public ItemReader<String> read() {
return new ListItemReader<>(Arrays.asList("java","spring","mybatis"));
}
@Bean
public JobExecutionListener getJobListener(){
return new MyJobListener();
}
}
在Job运行时可以以key=value形式传递参数,
Job执行的是Step,Job使用的参数肯定是在step中使用
我们只需要给step传递数据。 如何给step传递参数呢?
使用监听,使用step级别的监听传递数据
@Configuration
@EnableBatchProcessing
public class ParamtersDemo implements StepExecutionListener {
//注入创建任务对象的对象
@Autowired
private JobBuilderFactory jobBuilderFactory;
//注入创建step对象的对象
@Autowired
private StepBuilderFactory stepBuilderFactory;
private Map<String,JobParameter> parameters;
@Bean
public Job parameterJob(){
return jobBuilderFactory.get("parameterJob")
.start(parameterStep())
.build();
}
//Job执行的是Step,Job使用的参数肯定是在step中使用
//我们只需要给step传递数据。 如何给step传递参数呢?
//使用监听,使用step级别的监听传递数据
@Bean
public Step parameterStep() {
return stepBuilderFactory.get("parameterStep")
.listener(this)
.tasklet(new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
//输出接收到的参数的值
System.out.println(parameters.get("info"));
return RepeatStatus.FINISHED;
}
}).build();
}
@Override
public void beforeStep(StepExecution stepExecution) {
parameters = stepExecution.getJobParameters().getParameters();
}
@Override
public ExitStatus afterStep(StepExecution stepExecution) {
return null;
}
}
SpringBatch提供了JdbcPagingItemReader类
关键代码:
@Bean
@StepScope //指定范围,Step范围之内
public JdbcPagingItemReader<User> itemReaderFromDB() {
//创建返回值需要的对象
JdbcPagingItemReader<User> jdbcPagingItemReader = new JdbcPagingItemReader<>();
//指明数据源
jdbcPagingItemReader.setDataSource(dataSource);
//一次取几个数据
jdbcPagingItemReader.setFetchSize(2);
//读写转化为User对象
jdbcPagingItemReader.setRowMapper(new RowMapper<User>() {
@Override
public User mapRow(ResultSet resultSet, int i) throws SQLException {
//最终得到的是一个User对象
User user = new User();
user.setId(resultSet.getInt(1));
user.setName(resultSet.getString(2));
user.setAge(resultSet.getInt(3));
return user;
}
});
//sql语句
MySqlPagingQueryProvider provider = new MySqlPagingQueryProvider();
provider.setSelectClause("id,name,age");//指定查询字段
provider.setFromClause("from user");//指定查询表
//根据什么排序
Map<String, Order> map = new HashMap<>(1);//指定初始大小
map.put("id", Order.ASCENDING);
provider.setSortKeys(map);
jdbcPagingItemReader.setQueryProvider(provider);
return jdbcPagingItemReader;
}
FlatFileItemReader类
关键代码:
@Bean
@StepScope
public FlatFileItemReader<? extends Customer> flatFileItemReader() {
//创建返回对象
FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
//数据据
reader.setResource(new ClassPathResource("customer.txt"));
//因为第一行是字段名,所以跳过第一行
reader.setLinesToSkip(1);
//解析数据
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
//指明四个字段
tokenizer.setNames(new String[]{"id","name","birthday"});
//把解析出的一行数据映射为一个Customer
DefaultLineMapper<Customer> mapper = new DefaultLineMapper<>();
mapper.setLineTokenizer(tokenizer);
mapper.setFieldSetMapper(new FieldSetMapper<Customer>() {
@Override
public Customer mapFieldSet(FieldSet fieldSet) throws BindException {
Customer customer = new Customer();
customer.setId(fieldSet.readLong("id"));
customer.setName(fieldSet.readString("name"));
customer.setBirthday(fieldSet.readString("birthday"));
return customer;
}
});
//做一下检查
mapper.afterPropertiesSet();
reader.setLineMapper(mapper);//行内映射
return reader;
}
StaxEventItemReader类
将xml对象转化为实体对象要依赖的一些包如下:
<dependency>
<groupId>org.springframeworkgroupId>
<artifactId>spring-oxmartifactId>
dependency>
<dependency>
<groupId>com.thoughtworks.xstreamgroupId>
<artifactId>xstreamartifactId>
<version>1.4.11.1version>
dependency>
关键代码:
@Bean
@StepScope
public StaxEventItemReader<Customer> xmlFileReader() {
StaxEventItemReader<Customer> reader = new StaxEventItemReader<>();
reader.setResource(new ClassPathResource("customer.xml"));
//指定需要处理的根标签
reader.setFragmentRootElementName("customer");
//把xml转化成对象
XStreamMarshaller unmarshaller = new XStreamMarshaller();
HashMap<String, Class> map = new HashMap<>();
map.put("customer",Customer.class);
unmarshaller.setAliases(map);
reader.setUnmarshaller(unmarshaller);
return reader;
}
MultiResourceItemReader类
也是按照一个文件一个文件读的方式
关键代码:
@Value("classpath:/customer*.txt")
private Resource[] fileResources;
@Bean
@StepScope
public MultiResourceItemReader<Customer> multiFileReader() {
//创建返回对象
MultiResourceItemReader<Customer> reader = new MultiResourceItemReader<>();
reader.setDelegate(flatFileItemReader());
reader.setResources(fileResources);
return reader;
}
@Bean
@StepScope
public FlatFileItemReader<? extends Customer> flatFileItemReader() {
FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
reader.setResource(new ClassPathResource("file.txt"));
//跳过第一行
// reader.setLinesToSkip(1);
//解析数据
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
tokenizer.setNames(new String[]{"id","name","birthday"});
//把解析出的一行数据映射为一个Customer
DefaultLineMapper<Customer> mapper = new DefaultLineMapper<>();
mapper.setLineTokenizer(tokenizer);
mapper.setFieldSetMapper(new FieldSetMapper<Customer>() {
@Override
public Customer mapFieldSet(FieldSet fieldSet) throws BindException {
Customer customer = new Customer();
customer.setId(fieldSet.readLong("id"));
customer.setName(fieldSet.readString("name"));
customer.setBirthday(fieldSet.readString("birthday"));
return customer;
}
});
mapper.afterPropertiesSet();
reader.setLineMapper(mapper);
return reader;
}
在读取数据的时候出现异常,可能会导致任务结束,SpringBatch提供了解决方案。
实现ItemStreamReader接口
当读取的时候出现异常会记录下出现异常的行数,在下次重启任务的时候级从这个地方开始读取
ItemReader是一个数据一个数据的读,而ItemWriter是一批一批的输出,chunk中的值就是批的量
@Component("myWriter")
public class MyWriter implements ItemWriter<String> {
@Override
public void write(List<? extends String> list) throws Exception {
System.out.println(list.size());//每批处理多少个数据
for (String s : list) {
System.out.println(s);
}
}
}
SpringBatch提供了很多输入到数据库中的类,写入MySQL中用JdbcBatchItemWriter来处理
@Configuration
public class ItemWriterDbConfig {
@Autowired
private DataSource dataSource;
@Bean
public JdbcBatchItemWriter<Customer> itemWriterDb(){
JdbcBatchItemWriter<Customer> writer = new JdbcBatchItemWriter<>();
//设置数据源
writer.setDataSource(dataSource);
//sql语句
writer.setSql("insert into customer(id,name,birthday) values" +
"(:id,:name,:birthday)");
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>());
return writer;
}
}
FlatFileItemWriter
@Configuration
public class FileItemWriterConfig {
@Bean
public FlatFileItemWriter<Customer> flatFileItemWriter() throws Exception {
//把Customer对象转化为字符串输出到文件
FlatFileItemWriter<Customer> writer = new FlatFileItemWriter<>();
String path = "D:\\code\\java\\springbatch-study\\itemwriterfile\\customer.txt";
writer.setResource(new FileSystemResource(path));
//把Customer对象抓换成字符串
writer.setLineAggregator(new LineAggregator<Customer>() {
ObjectMapper mapper = new ObjectMapper();
@Override
public String aggregate(Customer customer) {
String str = null;
try {
str = mapper.writeValueAsString(customer);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
return str;
}
});
writer.afterPropertiesSet();
return writer;
}
}
StaxEvenItemWriter
@Bean
public StaxEventItemWriter<Customer> xmlItemWriter() throws Exception {
StaxEventItemWriter<Customer> writer = new StaxEventItemWriter<>();
//把对象转化为xml
XStreamMarshaller marshaller = new XStreamMarshaller();
Map<String,Class> aliases = new HashMap<>();
aliases.put("customer", Customer.class);
marshaller.setAliases(aliases);
writer.setRootTagName("customers");
writer.setMarshaller(marshaller);
String path = "D:\\code\\java\\springbatch-study\\itemwriterxml\\src\\main\\resources\\customer.xml";
writer.setResource(new FileSystemResource(path));
writer.afterPropertiesSet();
return writer;
}
CompositeItemWriter 实现数据输出到多个文件
ClassifierCompositeItemWriter 对数据进行分离,根据分类写入到不同的文件
/**
* 输出数据到多个文件,只负责调用
* @return
* @throws Exception
*/
@Bean
public CompositeItemWriter<Customer> multiFileItemWriter() throws Exception {
CompositeItemWriter<Customer> writer = new CompositeItemWriter<>();
//指定谁来输出
writer.setDelegates(Arrays.asList(flatFileItemWriter(),xmlItemWriter()));
//检查一下
writer.afterPropertiesSet();
return writer;
}
/**
* 数据实现分类
*/
@Bean
public ClassifierCompositeItemWriter<Customer> multiFileItemWriter(){
ClassifierCompositeItemWriter<Customer> writer = new ClassifierCompositeItemWriter<>();
//实现分离的方法
writer.setClassifier(new Classifier<Customer, ItemWriter<? super Customer>>() {
@Override
public ItemWriter<? super Customer> classify(Customer customer) {
//按照customer的id进行分类
ItemWriter<Customer> write = null;
try {
write = customer.getId()%2==0?jsonFileItemWriter():xmlItemWriter();
} catch (Exception e) {
e.printStackTrace();
}
return write;
}
});
return writer;
}
@Autowired
@Qualifier("jsonFileItemWriter")
private ItemStreamWriter<? super Customer> jsonFileItemWriter;
@Autowired
@Qualifier("xmlItemWriter")
private ItemStreamWriter<? super Customer> xmlItemWriter;
@Bean
public Step multiFileItemWriterDemoStep() {
return stepBuilderFactory.get("multiFileItemWriterDemoStep")
.<Customer,Customer>chunk(5)
.reader(dbJdbcReader)
.writer(multiFileItemWriter)
.stream(jsonFileItemWriter)
.stream(xmlItemWriter)
.build();
}
ItemProcessor用于处理业务逻辑,验证,过滤等功能,
CompositeItemProcessor
/**
* 只返回id为偶数的customer
*/
@Component
public class IdFilterProcessor implements ItemProcessor<Customer,Customer> {
@Override
public Customer process(Customer customer) throws Exception {
if (customer.getId()%2==0){
return customer;
}
return null; //相当于把该对象过滤掉
}
}
@Bean
public Step itemProcessorDemo12Step() {
return stepBuilderFactory.get("itemProcessorDemo12Step")
.<Customer,Customer>chunk(3)
.reader(flatFileItemReader())
.processor(nameUpperProcessor) //只使用一个itemProcessor时
.writer(flatFileItemWriter)
.build();
}
@Bean
public Step itemProcessorDemo12Step() {
return stepBuilderFactory.get("itemProcessorDemo12Step")
.<Customer,Customer>chunk(3)
.reader(flatFileItemReader())
// .processor(nameUpperProcessor) //只使用一个itemProcessor时
.processor(process()) //使用多种itemProcessor时
.writer(flatFileItemWriter)
.build();
}
//有多种处理数据的方式
@Bean
public CompositeItemProcessor<Customer,Customer> process(){
CompositeItemProcessor<Customer, Customer> processor = new CompositeItemProcessor<>();
List<ItemProcessor<Customer,Customer>> delegates = new ArrayList<>();
delegates.add(nameUpperProcessor);
delegates.add(idFilterProcessor);
processor.setDelegates(delegates);
return processor;
}
默认情况下,当任务出现异常时,SpringBatch会结束任务,当使用相同的参数重启任务时,SpringBatch会去执行剩余还没有执行的任务。
测试关键代码如下:
@Configuration
@EnableBatchProcessing
public class ErrorDemo {
@Autowired
private JobBuilderFactory jobBuilderFactory;
@Autowired
private StepBuilderFactory stepBuilderFactory;
@Bean
public Job errorDemoJob1(){
return jobBuilderFactory.get("errorDemoJob1")
.start(errorStep11())
.next(errorStep21())
.build();
}
@Bean
public Step errorStep11() {
return stepBuilderFactory.get("errorStep11")
.tasklet(errorHandling())
.build();
}
@Bean
public Step errorStep21() {
return stepBuilderFactory.get("errorStep21")
.tasklet(errorHandling())
.build();
}
@Bean
@StepScope
public Tasklet errorHandling() {
return new Tasklet() {
@Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
Map<String, Object> stepExecutionContext = chunkContext.getStepContext().getStepExecutionContext();
if (stepExecutionContext.containsKey("tqq")) {
System.out.println("The second run will success");
return RepeatStatus.FINISHED;
} else {
System.out.println("The first run will fail");
chunkContext.getStepContext().getStepExecution().getExecutionContext().put("tqq", true);
throw new RuntimeException("error ...");
}
}
};
}
}
当任务在执行过程中出现错误时,我我们不希望任务被停止,而是希望他能够重试
容错
容什么错
重试多少次
@Bean
public Step retryDemoStep() {
return stepBuilderFactory.get("retryDemoStep")
.<String,String>chunk(5)
.reader(reader())
.processor(retryItemProcessor)
.writer(retryItemWriter)
.faultTolerant()//容错
.retry(CustomRetryException.class)//容什么错,发生什么错的时候进行重试
.retryLimit(5)//重试多少次
.build();
}
当任务出现异常的时候跳过,继续处理后面的任务
(步骤和错误重试差不多)
跳过,
跳过什么错
最多跳过几次
@Bean
public Step skipDemoStep() {
return stepBuilderFactory.get("skipDemoStep")
.<String,String>chunk(10)
.reader(reader())
.processor(skipItemProcessor)
.writer(skipItemWriter)
.faultTolerant()//容错
.skip(CustomRetryException.class)//容什么错,发生什么错的时候进行跳过
.skipLimit(5)//跳过多少次
.build();
}
在任务出现异常的时候选择了跳过,要记录跳过的错误的信息,此时就用错误跳过监听器
@Bean
public Step skipListenerDemoStep() {
return stepBuilderFactory.get("skipListenerDemoStep")
.<String,String>chunk(10)
.reader(reader())
.processor(skipItemProcessor)
.writer(skipItemWriter)
.faultTolerant()//容错
.skip(CustomRetryException.class)//容什么错,发生什么错的时候进行跳过
.skipLimit(5)//跳过多少次
.listener(mySkipListener)
.build();
}
监听记录
@Component
public class MySkipListener implements SkipListener<String,String> {
@Override
public void onSkipInRead(Throwable throwable) {
}
@Override
public void onSkipInWrite(String s, Throwable throwable) {
}
@Override
public void onSkipInProcess(String s, Throwable throwable) {
System.out.println(s+"在处理过程中出现了异常:"+throwable);
}
}
控制任务什么时候启动
是对JobLaucher的封装,比JobOperator功能强大,但是使用起来更加复杂起来