序
本文主要介绍下chaos-monkey-spring-boot
Chaos Monkey for Spring Boot
chaos-monkey-spring-boot是专门为Spring Boot打造的Chaos Monkey
image
主要有有如下几个方面的Assaults
- Latency Assault
- Exception Assault
- AppKiller Assault
实例
maven
de.codecentric
chaos-monkey-spring-boot
1.0.1
配置
chaos.monkey.assaults.level=5
chaos.monkey.assaults.latencyRangeStart=10000
chaos.monkey.assaults.latencyRangeEnd=15000
chaos.monkey.assaults.latencyActive=true
chaos.monkey.assaults.exceptionsActive=true
chaos.monkey.assaults.killApplicationActive=true
chaos.monkey.watcher.controller=true
chaos.monkey.watcher.restController=true
chaos.monkey.watcher.service=true
chaos.monkey.watcher.repository=true
运行
2018-04-20 22:50:02.475 INFO 2861 --- [ main] d.c.s.b.c.monkey.component.ChaosMonkey :
_____ _ __ __ _
/ ____| | | \/ | | |
| | | |__ __ _ ___ ___ | \ / | ___ _ __ | | _____ _ _
| | | '_ \ / _` |/ _ \/ __| | |\/| |/ _ \| '_ \| |/ / _ | | | |
| |____| | | | (_| | (_) \__ \ | | | | (_) | | | | | __| |_| |
\_____|_| |_|\__,_|\___/|___/ |_| |_|\___/|_| |_|_|\_\___|\__, |
__/ |
_ready to do evil! |___/
:: Chaos Monkey for Spring Boot ::
使用-Dspring.profiles.active=chaos-monkey
Assaluts输出
Latency Assault
2018-04-20 22:37:17.373 INFO 2827 --- [nio-8080-exec-9] d.c.s.b.c.monkey.component.ChaosMonkey : Chaos Monkey - timeout
Exception Assault
2018-04-20 22:37:01.379 ERROR 2827 --- [nio-8080-exec-5] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.RuntimeException: Chaos Monkey - RuntimeException] with root cause
java.lang.RuntimeException: Chaos Monkey - RuntimeException
at de.codecentric.spring.boot.chaos.monkey.component.ChaosMonkey.generateChaosException(ChaosMonkey.java:76) ~[chaos-monkey-spring-boot-1.0.1.jar:na]
at de.codecentric.spring.boot.chaos.monkey.component.ChaosMonkey.callChaosMonkey(ChaosMonkey.java:45) ~[chaos-monkey-spring-boot-1.0.1.jar:na]
at de.codecentric.spring.boot.chaos.monkey.watcher.SpringRestControllerAspect.intercept(SpringRestControllerAspect.java:37) ~[chaos-monkey-spring-boot-1.0.1.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_71]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_71]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_71]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_71]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at com.example.controller.HelloController$$EnhancerBySpringCGLIB$$1390f753.hello() ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_71]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_71]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_71]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_71]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:84) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.29.jar:8.5.29]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.29.jar:8.5.29]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_71]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.29.jar:8.5.29]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_71]
AppKiller Assault
2018-04-20 22:51:08.271 INFO 2861 --- [nio-8080-exec-3] d.c.s.b.c.monkey.component.ChaosMonkey : Chaos Monkey - I am killing your Application!
2018-04-20 22:51:08.272 INFO 2861 --- [nio-8080-exec-3] ConfigServletWebServerApplicationContext : Closing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2b6856dd: startup date [Fri Apr 20 22:49:58 CST 2018]; root of context hierarchy
2018-04-20 22:51:08.275 INFO 2861 --- [nio-8080-exec-3] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
源码解析
ChaosMonkeyConfiguration
chaos-monkey-spring-boot-1.0.1-sources.jar!/de/codecentric/spring/boot/chaos/monkey/configuration/ChaosMonkeyConfiguration.java
@Configuration
@Profile("chaos-monkey")
@EnableConfigurationProperties({AssaultProperties.class, WatcherProperties.class})
@Import(EndpointConfiguration.class)
public class ChaosMonkeyConfiguration {
private static final Logger LOGGER = LoggerFactory.getLogger(ChaosMonkey.class);
private final WatcherProperties watcherProperties;
private final AssaultProperties assaultProperties;
public ChaosMonkeyConfiguration(WatcherProperties watcherProperties, AssaultProperties assaultProperties) {
this.watcherProperties = watcherProperties;
this.assaultProperties = assaultProperties;
try {
String chaosLogo = StreamUtils.copyToString(new ClassPathResource("chaos-logo.txt").getInputStream(), Charset.defaultCharset());
LOGGER.info(chaosLogo);
} catch (IOException e) {
LOGGER.info("Chaos Monkey - ready to do evil");
}
}
@Bean
public ChaosMonkeySettings settings() {
return new ChaosMonkeySettings(assaultProperties, watcherProperties);
}
@Bean
public ChaosMonkey chaosMonkey() {
return new ChaosMonkey(assaultProperties);
}
@Bean
@Conditional(AttackControllerCondition.class)
public SpringControllerAspect controllerAspect() {
return new SpringControllerAspect(chaosMonkey());
}
@Bean
@Conditional(AttackRestControllerCondition.class)
public SpringRestControllerAspect restControllerAspect() {
return new SpringRestControllerAspect(chaosMonkey());
}
@Bean
@Conditional(AttackServiceCondition.class)
public SpringServiceAspect serviceAspect() {
return new SpringServiceAspect(chaosMonkey());
}
}
这里加载了配置，然后实例化了ChaosMonkey，以及根据条件实例化Controller，RestController，Service注解的aspect，其原理就是根据aspect进行拦截处理，产生相应的chaos
ChaosMonkey
chaos-monkey-spring-boot-1.0.1-sources.jar!/de/codecentric/spring/boot/chaos/monkey/component/ChaosMonkey.java
public void callChaosMonkey() {
if (isTrouble()) {
int exceptionRand = assaultProperties.getExceptionRandom();
if (assaultProperties.isLatencyActive() && assaultProperties.isExceptionsActive()) {
// Timeout or Exception?
if (exceptionRand < 7) {
generateLatency();
} else {
generateChaosException();
}
} else if (assaultProperties.isLatencyActive()) {
generateLatency();
} else if (assaultProperties.isExceptionsActive()) {
generateChaosException();
} else if (assaultProperties.isKillApplicationActive()) {
killTheBossApp();
}
}
}
ChaosMonkey主要提供了generateLatency、generateChaosException、killTheBossApp这几个方法来分别实现Latency Assault、Exception Assault、AppKiller Assault
Aspect实例
chaos-monkey-spring-boot-1.0.1-sources.jar!/de/codecentric/spring/boot/chaos/monkey/watcher/SpringRestControllerAspect.java
@Aspect
public class SpringRestControllerAspect {
private static final Logger LOGGER = LoggerFactory.getLogger(SpringRestControllerAspect.class);
private final ChaosMonkey chaosMonkey;
public SpringRestControllerAspect(ChaosMonkey chaosMonkey) {
this.chaosMonkey = chaosMonkey;
}
@Pointcut("within(@org.springframework.web.bind.annotation.RestController *)")
public void classAnnotatedWithControllerPointcut() {
}
@Pointcut("execution(* *.*(..))")
public void allPublicMethodPointcut() {
}
@Around("classAnnotatedWithControllerPointcut() && allPublicMethodPointcut()")
public Object intercept(ProceedingJoinPoint pjp) throws Throwable {
LOGGER.debug(LOGGER.isDebugEnabled() ? "RestController class and public method detected: " + pjp.getSignature() : null);
chaosMonkey.callChaosMonkey();
return pjp.proceed();
}
}
可以看到这里使用@Around拦截，然后调用chaosMonkey.callChaosMonkey()总的入口方法，由它判断是否需要产生chaos
小结
chaos-monkey-spring-boot是个好东东，非常适合用来进行故障演练，暴露服务间调用的问题，好提升系统的健壮性、故障自动恢复能力等。
doc
- Chaos工程
- Twilio的混沌工程实践
- chaos-monkey-spring-boot
- PRINCIPLES OF CHAOS ENGINEERING
- Chaos Engineering的历史、原则以及实践