Spring Cloud 学习--Hystrix应用

上一篇介绍了Hystrix基本功能和单独使用的方式，今天继续学习如何将Hystrix融入SpringCloud组件中去。

在Ribbon上使用熔断器

在 pom.xml 文件中引入 hystrix 的 依赖spring-cloud-starter-hystrix：

   org.springframework.cloud
   spring-cloud-starter-hystrix

在应用的启动类上使用 @EnableHystrix 开启 hystrix 的功能。

package com.rickiyang.learn;

import com.netflix.loadbalancer.IRule;
import com.netflix.loadbalancer.RandomRule;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.cloud.netflix.hystrix.EnableHystrix;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@EnableHystrix
@EnableDiscoveryClient
@SpringBootApplication
public class RibbonDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(RibbonDemoApplication.class, args);
    }

    @Bean
    @LoadBalanced
    RestTemplate restTemplate() {
        return new RestTemplate();
    }

    @Bean
    public IRule ribbonRule() {
        return new RandomRule();//这里配置策略，和配置文件对应
    }
}

使用注解 @HystrixCommand 标记调用失败时需要熔断的方法，fallbackMethod 属性指定 降级方法 的 方法名 为 fallback。

package com.rickiyang.learn.service;

import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand;
import com.rickiyang.learn.entity.Person;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestTemplate;

/**
 * @author rickiyang
 * @date 2019-10-08
 * @Desc TODO
 */
@Service
public class DemoService {

    @Autowired
    RestTemplate restTemplate;

    /**
     * 支持服务降级
     * 自定义降级处理的超时时间，并发数
     * 自定义执行该任务的线程池参数
     * 自定义执行熔断逻辑的异常
     * @param name
     * @return
     */
    @HystrixCommand(
        fallbackMethod = "hiError",
        commandProperties={
            // 降级处理超时时间设置
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000"),
            // 任意时间点允许的最高并发数。超过该设置值后，拒绝执行请求。
            @HystrixProperty(name = "fallback.isolation.semaphore.maxConcurrentRequests", value = "1000"),
        },
        // 配置执行的线程池
        threadPoolProperties = {
            @HystrixProperty(name = "coreSize", value = "20"),
            @HystrixProperty(name = "maxQueueSize", value = "-1"),
        },
        // 该异常不执行熔断，去执行该异常抛出的自己逻辑
        ignoreExceptions = {NoSuchMethodException.class}
    )
    public String hello(String name) {
        return restTemplate.getForEntity("http://eureka-client/hello/" + name, String.class).getBody();
    }

    public String fail(String name) {
        return "accu"+name+",error!";
    }

}

在Feign上使用熔断器

Feign 是自带 断路器 的，不过需要在 配置文件 中开启 hystrix 的配置。

feign:
  hystrix:
    enabled: true

Hystrix 支持 降级回退 操作，当 发生熔断 或 出现错误 时，调用会执行 默认代码路径。

package com.rickiyang.learn.service;

import com.rickiyang.learn.entity.Person;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.*;

/**
 * @author: rickiyang
 * @date: 2019/10/5
 * @description:
 */
@FeignClient(name= "eureka-client",fallback = HelloFailBackService.class)
public interface HelloRemote {


    @RequestMapping(value = "/hello/{name}")
    String hello(@PathVariable(value = "name") String name);


    @PostMapping(value ="/add",produces = "application/json; charset=UTF-8")
    String addPerson(@RequestBody Person person);

    @GetMapping("/getPerson/{id}")
    String getPerson(@PathVariable("id") Integer id);

}

通过设置 fallback 属性为实现 降级回退 的 类，来启用 @FeignClient 的 失败降级。

package com.rickiyang.learn.service;

import com.rickiyang.learn.entity.Person;

public class HelloFailBackService implements HelloRemote {

    @Override
    public String hello(String name) {
        return "";
    }

    @Override
    public String addPerson(Person person) {
        return null;
    }

    @Override
    public String getPerson(Integer id) {
        return null;
    }
}

如果需要获取导致 回退触发 的原因，可以指定 @FeignClient 注解内部的 fallbackFactory 属性，fallbackFactory 属性和 fallback 属性不能一起使用。

package com.rickiyang.learn.service;

import com.rickiyang.learn.entity.Person;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.*;

/**
 * @author: rickiyang
 * @date: 2019/10/5
 * @description:
 */
@FeignClient(name= "eureka-client",fallback = HelloFailBackFacgtory.class)
public inte rface HelloRemote {

    @RequestMapping(value = "/hello/{name}")
    String hello(@PathVariable(value = "name") String name);


    @PostMapping(value ="/add",produces = "application/json; charset=UTF-8")
    String addPerson(@RequestBody Person person);

    @GetMapping("/getPerson/{id}")
    String getPerson(@PathVariable("id") Integer id);

}

然后提供一个 FallbackFactory 的 实现类，实现类指定 泛型参数 为 HelloService。

package com.rickiyang.learn.service;

import com.rickiyang.learn.entity.Person;
import feign.hystrix.FallbackFactory;

public class HelloFailBackFacgtory implements FallbackFactory {

    @Override
    public HelloRemote create(Throwable throwable) {
        return new HelloRemote() {
            @Override
            public String hello(String name) {
                return "fail reason is : " + throwable.getMessage();
            }

            @Override
            public String addPerson(Person person) {
                return "fail reason is : " + throwable.getMessage();

            }

            @Override
            public String getPerson(Integer id) {
                return "fail reason is : " + throwable.getMessage();

            }
        };
    }
}

Hystrix Dashboard监控熔断器的状态

Hystrix Dashboard 是一个 监控熔断器 状况的组件，提供了 数据监控 和 图形界面。

在Ribbon中使用Hystrix Dashboard

在加入 spring-cloud-starter-hystrix 依赖的基础上，加入下面的依赖：

    org.springframework.boot
    spring-boot-starter-actuator


    org.springframework.cloud
    spring-cloud-starter-hystrix-dashboard

在应用程序 启动类 已经加上 @EnableHystrix 的基础上，加入 @EnableHystrixDashboard 注解，代码如下：

package com.rickiyang.learn;

import com.netflix.loadbalancer.IRule;
import com.netflix.loadbalancer.RandomRule;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.cloud.netflix.hystrix.EnableHystrix;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@EnableHystrix
@EnableHystrixDashboard
@EnableDiscoveryClient
@SpringBootApplication
public class RibbonDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(RibbonDemoApplication.class, args);
    }

    @Bean
    @LoadBalanced
    RestTemplate restTemplate() {
        return new RestTemplate();
    }

    @Bean
    public IRule ribbonRule() {
        return new RandomRule();//这里配置策略，和配置文件对应
    }
}

Hystrix源码分析

前文提到了Hystrix核心功能包括：隔离机制，熔断机制，降级机制。围绕着这3点我们一起看一下它们分别是如何实现的。
首先从 @EnableHystrix入手，看一下开启Hystrix功能的时候都做了什么。

package org.springframework.cloud.netflix.hystrix;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@EnableCircuitBreaker
public @interface EnableHystrix {

}

该注解的功能是为了引用@EnableCircuitBreaker注解。

package org.springframework.cloud.client.circuitbreaker;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import org.springframework.context.annotation.Import;

/**
 * Annotation to enable a CircuitBreaker implementation.
 * http://martinfowler.com/bliki/CircuitBreaker.html
 * @author Spencer Gibb
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Import(EnableCircuitBreakerImportSelector.class)
public @interface EnableCircuitBreaker {

}

这里使用了import注解引入EnableCircuitBreakerImportSelector：

@Order(Ordered.LOWEST_PRECEDENCE - 100)
public class EnableCircuitBreakerImportSelector extends
        SpringFactoryImportSelector {

    @Override
    protected boolean isEnabled() {
        return getEnvironment().getProperty(
                "spring.cloud.circuit.breaker.enabled", Boolean.class, Boolean.TRUE);
    }

}

作用是将spring.cloud.circuit.breaker.enabled设置为启用。那么在进行配置扫描的时候就会按照启用的配置去进行相应的操作。

再回到EnableCircuitBreaker，查看被引用的位置，能看到在spring-cloud-netflix-core包的配置文件中有一个引用：

org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker=\
org.springframework.cloud.netflix.hystrix.HystrixCircuitBreakerConfiguration

将 EnableCircuitBreaker 指向了 HystrixCircuitBreakerConfiguration 类：

@Configuration
public class HystrixCircuitBreakerConfiguration {

    @Bean
    public HystrixCommandAspect hystrixCommandAspect() {
        return new HystrixCommandAspect();
    }

    @Bean
    public HystrixShutdownHook hystrixShutdownHook() {
        return new HystrixShutdownHook();
    }

    @Bean
    public HasFeatures hystrixFeature() {
        return HasFeatures.namedFeatures(new NamedFeature("Hystrix", HystrixCommandAspect.class));
    }


    /**
     * {@link DisposableBean} that makes sure that Hystrix internal state is cleared when
     * {@link ApplicationContext} shuts down.
     */
    private class HystrixShutdownHook implements DisposableBean {

        @Override
        public void destroy() throws Exception {
            // Just call Hystrix to reset thread pool etc.
            Hystrix.reset();
        }

    }

}

这里有一个关键的实例： HystrixCommandAspect ，从名称就能看出是扫描 HystrixCommand 注解的切面。

在 HystrixCommandAspect 定义了两个切面：

public class HystrixCommandAspect {

    private static final Map META_HOLDER_FACTORY_MAP;

    static {
        META_HOLDER_FACTORY_MAP = ImmutableMap.builder()
                .put(HystrixPointcutType.COMMAND, new CommandMetaHolderFactory())
                .put(HystrixPointcutType.COLLAPSER, new CollapserMetaHolderFactory())
                .build();
    }

    @Pointcut("@annotation(com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand)")

    public void hystrixCommandAnnotationPointcut() {
    }

    @Pointcut("@annotation(com.netflix.hystrix.contrib.javanica.annotation.HystrixCollapser)")
    public void hystrixCollapserAnnotationPointcut() {
    }
  
  @Around("hystrixCommandAnnotationPointcut() || hystrixCollapserAnnotationPointcut()")
    public Object methodsAnnotatedWithHystrixCommand(final ProceedingJoinPoint joinPoint) throws Throwable {
      //拿到切面声明的方法
        Method method = getMethodFromTarget(joinPoint);
        Validate.notNull(method, "failed to get method from joinPoint: %s", joinPoint);
      //如果方法的注解不是这两个不进入
        if (method.isAnnotationPresent(HystrixCommand.class) && method.isAnnotationPresent(HystrixCollapser.class)) {
            throw new IllegalStateException("method cannot be annotated with HystrixCommand and HystrixCollapser " +
                    "annotations at the same time");
        }
      //从工厂对象中取出当前注解所对应的执行逻辑工厂类
        MetaHolderFactory metaHolderFactory = META_HOLDER_FACTORY_MAP.get(HystrixPointcutType.of(method));
        MetaHolder metaHolder = metaHolderFactory.create(joinPoint);
      //执行工厂类逻辑
        HystrixInvokable invokable = HystrixCommandFactory.getInstance().create(metaHolder);
        ExecutionType executionType = metaHolder.isCollapserAnnotationPresent() ?
                metaHolder.getCollapserExecutionType() : metaHolder.getExecutionType();

        Object result;
        try {
          //这里区分是要同步执行还是异步执行
            if (!metaHolder.isObservable()) {
                result = CommandExecutor.execute(invokable, executionType, metaHolder);
            } else {
                result = executeObservable(invokable, executionType, metaHolder);
            }
        } catch (HystrixBadRequestException e) {
            throw e.getCause() != null ? e.getCause() : e;
        } catch (HystrixRuntimeException e) {
            throw hystrixRuntimeExceptionToThrowable(metaHolder, e);
        }
        return result;
    }

  ......
  ......
  ......
  
}

这里会扫描两个注解：

• HystrixCommand
• HystrixCollapser

methodsAnnotatedWithHystrixCommand()方法使用环绕通知拦截所有@HystrixCommand 和 @HystrixCollapser注解的方法。

这里用到的设计模式还是挺多：

1. 首先构造了一个分别执行扫描到上面两个注解的方法工厂：CommandMetaHolderFactory，通过工厂类来决定执行哪个注解对应的逻辑，然后把对应的逻辑封装成MetaHolder；
2. 接着会使用HystrixCommandFactory工厂将每一个被扫描的方法统一封装成HystrixInvokable对象；
3. 然后去执行对每一个被拦截的方法生成相应拦截配置的逻辑。

关于hystrix初始化的过程我们暂时就说这么多，了解到在哪里执行了注解扫描去做了初始化，下面我们分别详述各个功能在Hystrix中是如何实现。

隔离机制

Hystrix可以指定为每一个请求创建独立的线程池来执行，首先看一下@HystrixCommand的参数说明：

public @interface HystrixCommand {
  // HystrixCommand 命令所属的组的名称：默认注解方法类的名称
  String groupKey() default "";

  // HystrixCommand 命令的key值，默认值为注解方法的名称
  String commandKey() default "";

  // 线程池名称，默认定义为groupKey
  String threadPoolKey() default "";
  // 定义回退方法的名称, 此方法必须和hystrix的执行方法在相同类中
  String fallbackMethod() default "";
  // 配置hystrix命令的参数
  HystrixProperty[] commandProperties() default {};
  // 配置hystrix依赖的线程池的参数
  HystrixProperty[] threadPoolProperties() default {};

  // 如果hystrix方法抛出的异常包括RUNTIME_EXCEPTION，则会被封装HystrixRuntimeException异常。我们也可以通过此方法定义哪些需要忽略的异常
  Class[] ignoreExceptions() default {};

  // 定义执行hystrix observable的命令的模式，类型详细见ObservableExecutionMode
  ObservableExecutionMode observableExecutionMode() default ObservableExecutionMode.EAGER;

  // 如果hystrix方法抛出的异常包括RUNTIME_EXCEPTION，则会被封装HystrixRuntimeException异常。此方法定义需要抛出的异常
  HystrixException[] raiseHystrixExceptions() default {};

  // 定义回调方法：但是defaultFallback不能传入参数，返回参数和hystrix的命令兼容
  String defaultFallback() default "";
}

threadPoolKey（）可以指定线程池名称。

还记得上一篇中我们讲到 HystrixCommand / HystrixObservableCommand类，被以来的服务想要被Hystrix封装，只用继承这两个类中的一个即可。只是现在使用了 @HystrixCommand注解将这一部分逻辑封装了你无法看到。

首先我们从HystrixCommand类入手，可以看到它继承了 AbstractCommand，一般抽象类都会先默默地做一些事情为子类分担忧愁。我们看一下 AbstractCommand 中的逻辑：

protected AbstractCommand(HystrixCommandGroupKey group, HystrixCommandKey key, HystrixThreadPoolKey threadPoolKey, HystrixCircuitBreaker circuitBreaker, HystrixThreadPool threadPool,
                          HystrixCommandProperties.Setter commandPropertiesDefaults, HystrixThreadPoolProperties.Setter threadPoolPropertiesDefaults,
                          HystrixCommandMetrics metrics, TryableSemaphore fallbackSemaphore, TryableSemaphore executionSemaphore,
                          HystrixPropertiesStrategy propertiesStrategy, HystrixCommandExecutionHook executionHook) {

  this.commandGroup = initGroupKey(group);
  this.commandKey = initCommandKey(key, getClass());
  this.properties = initCommandProperties(this.commandKey, propertiesStrategy, commandPropertiesDefaults);
  //线程池的key
  this.threadPoolKey = initThreadPoolKey(threadPoolKey, this.commandGroup, this.properties.executionIsolationThreadPoolKeyOverride().get());
  this.metrics = initMetrics(metrics, this.commandGroup, this.threadPoolKey, this.commandKey, this.properties);
  this.circuitBreaker = initCircuitBreaker(this.properties.circuitBreakerEnabled().get(), circuitBreaker, this.commandGroup, this.commandKey, this.properties, this.metrics);
  //初始化线程池
  this.threadPool = initThreadPool(threadPool, this.threadPoolKey, threadPoolPropertiesDefaults);

  //Strategies from plugins
  this.eventNotifier = HystrixPlugins.getInstance().getEventNotifier();
  this.concurrencyStrategy = HystrixPlugins.getInstance().getConcurrencyStrategy();
  HystrixMetricsPublisherFactory.createOrRetrievePublisherForCommand(this.commandKey, this.commandGroup, this.metrics, this.circuitBreaker, this.properties);
  this.executionHook = initExecutionHook(executionHook);

  this.requestCache = HystrixRequestCache.getInstance(this.commandKey, this.concurrencyStrategy);
  this.currentRequestLog = initRequestLog(this.properties.requestLogEnabled().get(), this.concurrencyStrategy);

  /* fallback semaphore override if applicable */
  this.fallbackSemaphoreOverride = fallbackSemaphore;

  /* execution semaphore override if applicable */
  this.executionSemaphoreOverride = executionSemaphore;
}

直接进入到线程池初始化的代码：

/*
 * 初始化线程池的key
 * 如果key为空，默认使用HystrixCommandGroup的名称作为key
 *
 */
private static HystrixThreadPoolKey initThreadPoolKey(HystrixThreadPoolKey threadPoolKey, HystrixCommandGroupKey groupKey, String threadPoolKeyOverride) {
  if (threadPoolKeyOverride == null) {
    // we don't have a property overriding the value so use either HystrixThreadPoolKey or HystrixCommandGroup
    if (threadPoolKey == null) {
      /* use HystrixCommandGroup if HystrixThreadPoolKey is null */
      return HystrixThreadPoolKey.Factory.asKey(groupKey.name());
    } else {
      return threadPoolKey;
    }
  } else {
    // we have a property defining the thread-pool so use it instead
    return HystrixThreadPoolKey.Factory.asKey(threadPoolKeyOverride);
  }
}


/*
 * 初始化线程池
 * HystrixThreadPool 中构造了一个ConcurrentHashMap来保存所有的线程池
 *
 */
private static HystrixThreadPool initThreadPool(HystrixThreadPool fromConstructor, HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolProperties.Setter threadPoolPropertiesDefaults) {
  if (fromConstructor == null) {
    // get the default implementation of HystrixThreadPool
    return HystrixThreadPool.Factory.getInstance(threadPoolKey, threadPoolPropertiesDefaults);
  } else {
    return fromConstructor;
  }
}

/**
*
*从map中获取线程池，如果不存在则构造一个线程池对象存入
*/
static HystrixThreadPool getInstance(HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolProperties.Setter propertiesBuilder) {
  // get the key to use instead of using the object itself so that if people forget to implement equals/hashcode things will still work
  String key = threadPoolKey.name();

  // this should find it for all but the first time
  HystrixThreadPool previouslyCached = threadPools.get(key);
  if (previouslyCached != null) {
    return previouslyCached;
  }

  // 加锁 保证单机并发的安全性
  synchronized (HystrixThreadPool.class) {
    if (!threadPools.containsKey(key)) {
      //通过HystrixThreadPoolDefault类来构造线程池
      threadPools.put(key, new HystrixThreadPoolDefault(threadPoolKey, propertiesBuilder));
    }
  }
  return threadPools.get(key);
}

构造线程池的代码主要在 HystrixThreadPoolDefault 类中：

static class HystrixThreadPoolDefault implements HystrixThreadPool {
  
  public HystrixThreadPoolDefault(HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolProperties.Setter propertiesDefaults) {
    this.properties = HystrixPropertiesFactory.getThreadPoolProperties(threadPoolKey, propertiesDefaults);
    HystrixConcurrencyStrategy concurrencyStrategy = HystrixPlugins.getInstance().getConcurrencyStrategy();
    this.queueSize = properties.maxQueueSize().get();

    this.metrics = HystrixThreadPoolMetrics.getInstance(threadPoolKey,
                                                        concurrencyStrategy.getThreadPool(threadPoolKey, properties),
                                                        properties);
    this.threadPool = this.metrics.getThreadPool();
    this.queue = this.threadPool.getQueue();

    /* strategy: HystrixMetricsPublisherThreadPool */
    HystrixMetricsPublisherFactory.createOrRetrievePublisherForThreadPool(threadPoolKey, this.metrics, this.properties);
  }
  ......
  ......
  ......
  
}

注意这里有一个策略模式：HystrixConcurrencyStrategy，声明了不同的加载线程池的策略。

具体加载线程池是在：concurrencyStrategy.getThreadPool(threadPoolKey, properties)方法中：

public ThreadPoolExecutor getThreadPool(final HystrixThreadPoolKey threadPoolKey, HystrixThreadPoolProperties threadPoolProperties) {
  final ThreadFactory threadFactory = getThreadFactory(threadPoolKey);

  final boolean allowMaximumSizeToDivergeFromCoreSize = threadPoolProperties.getAllowMaximumSizeToDivergeFromCoreSize().get();
  final int dynamicCoreSize = threadPoolProperties.coreSize().get();
  final int keepAliveTime = threadPoolProperties.keepAliveTimeMinutes().get();
  final int maxQueueSize = threadPoolProperties.maxQueueSize().get();
  final BlockingQueue workQueue = getBlockingQueue(maxQueueSize);

  if (allowMaximumSizeToDivergeFromCoreSize) {
    final int dynamicMaximumSize = threadPoolProperties.maximumSize().get();
    if (dynamicCoreSize > dynamicMaximumSize) {
      logger.error("Hystrix ThreadPool configuration at startup for : " + threadPoolKey.name() + " is trying to set coreSize = " +
                   dynamicCoreSize + " and maximumSize = " + dynamicMaximumSize + ".  Maximum size will be set to " +
                   dynamicCoreSize + ", the coreSize value, since it must be equal to or greater than the coreSize value");
      return new ThreadPoolExecutor(dynamicCoreSize, dynamicCoreSize, keepAliveTime, TimeUnit.MINUTES, workQueue, threadFactory);
    } else {
      return new ThreadPoolExecutor(dynamicCoreSize, dynamicMaximumSize, keepAliveTime, TimeUnit.MINUTES, workQueue, threadFactory);
    }
  } else {
    return new ThreadPoolExecutor(dynamicCoreSize, dynamicCoreSize, keepAliveTime, TimeUnit.MINUTES, workQueue, threadFactory);
  }
}

这里构造线程池的方式就是我们熟悉的：new ThreadPoolExecutor()。

至此隔离机制中的线程池隔离我们就弄清楚了，线程池是以HystrixCommandGroupKey进行划分的，不同的CommandGroup有不同的线程池来处理。

熔断机制

上一节我们写过开启熔断器的代码：

/**
 * 一段简单的使用HystrixCommand封装服务隔离调用的实例
 */
public class QueryOrderIdCommand extends HystrixCommand {
    private final static Logger logger = LoggerFactory.getLogger(QueryOrderIdCommand.class);
    private String orderId = "";

    /**
     * 构造函数中封装了一些参数设置
     * @param orderId
     */
    public QueryOrderIdCommand(String orderId) {
        super(HystrixCommand.Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("orderService"))
                .andCommandKey(HystrixCommandKey.Factory.asKey("queryByOrderId"))
                .andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
                        .withCircuitBreakerRequestVolumeThreshold(10)//至少有10个请求，熔断器才进行错误率的计算
                        .withCircuitBreakerSleepWindowInMilliseconds(5000)//熔断器中断请求5秒后会进入半打开状态,放部分流量过去重试
                        .withCircuitBreakerErrorThresholdPercentage(50)//错误率达到50开启熔断保护
                        )
              .andThreadPoolKey(HystrixThreadPoolKey.Factory.asKey("orderServicePool"))
                .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties
                        .Setter().withCoreSize(10)));
        this.orderId = orderId;
    }
    ......
    ......
    ......
}

上面使用 @HystrixCommand 注解进行熔断的用法：

@HystrixCommand(
  fallbackMethod = "hiError",
  commandProperties={
    // 降级处理超时时间设置
    @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000"),
    // 任意时间点允许的最高并发数。超过该设置值后，拒绝执行请求。
    @HystrixProperty(name = "fallback.isolation.semaphore.maxConcurrentRequests", value = "1000"),
  },
  // 配置执行的线程池
  threadPoolProperties = {
    @HystrixProperty(name = "coreSize", value = "20"),
    @HystrixProperty(name = "maxQueueSize", value = "-1"),
  },
  // 该异常不执行熔断，去执行该异常抛出的自己逻辑
  ignoreExceptions = {NoSuchMethodException.class}
)
public String hello(String name) {
  return restTemplate.getForEntity("http://eureka-client/hello/" + name, String.class).getBody();
}

仍旧是在AbstractCommand的构造函数中，有熔断器初始化的逻辑：

this.circuitBreaker = initCircuitBreaker(this.properties.circuitBreakerEnabled().get(), circuitBreaker, this.commandGroup, this.commandKey, this.properties, this.metrics);

/*
 *调用 HystrixCircuitBreaker 工厂类方法执行初始化
 *
 */
private static HystrixCircuitBreaker initCircuitBreaker(boolean enabled, HystrixCircuitBreaker fromConstructor,
                                                        HystrixCommandGroupKey groupKey, HystrixCommandKey commandKey,
                                                        HystrixCommandProperties properties, HystrixCommandMetrics metrics) {
  if (enabled) {
    if (fromConstructor == null) {
      // get the default implementation of HystrixCircuitBreaker
      return HystrixCircuitBreaker.Factory.getInstance(commandKey, groupKey, properties, metrics);
    } else {
      return fromConstructor;
    }
  } else {
    return new NoOpCircuitBreaker();
  }
}

同样，在熔断器的保存逻辑中，也是将所有的熔断器存储在本地Map：

public static HystrixCircuitBreaker getInstance(HystrixCommandKey key, HystrixCommandGroupKey group, HystrixCommandProperties properties, HystrixCommandMetrics metrics) {
  // this should find it for all but the first time
  HystrixCircuitBreaker previouslyCached = circuitBreakersByCommand.get(key.name());
  if (previouslyCached != null) {
    return previouslyCached;
  }

  HystrixCircuitBreaker cbForCommand = circuitBreakersByCommand.putIfAbsent(key.name(), new HystrixCircuitBreakerImpl(key, group, properties, metrics));
  if (cbForCommand == null) {
    return circuitBreakersByCommand.get(key.name());
    return cbForCommand;
  }
}

在putIfAbsent()方法中，保存的是一个 HystrixCircuitBreakerImpl 对象，这里定义的是断路器所有实现的逻辑状态。

断路器的状态分为3种：

enum Status {
  CLOSED, OPEN, HALF_OPEN;
}

• CLOSED关闭状态：允许流量通过。
• OPEN打开状态：不允许流量通过，即处于降级状态，走降级逻辑。
• HALF_OPEN半开状态：允许某些流量通过，并关注这些流量的结果，如果出现超时、异常等情况，将进入OPEN状态，如果成功，那么将进入CLOSED状态。

在构造函数初始化的时候做了监听 metrics 的HealthCountsStream信息的异步操作：

protected HystrixCircuitBreakerImpl(HystrixCommandKey key, HystrixCommandGroupKey commandGroup, final HystrixCommandProperties properties, HystrixCommandMetrics metrics) {
  this.properties = properties;
  this.metrics = metrics;

  //On a timer, this will set the circuit between OPEN/CLOSED as command executions occur
  Subscription s = subscribeToStream();
  activeSubscription.set(s);
}

private Subscription subscribeToStream() {
  //这里会在每次执行onNext()事件的时候来评估是否需要打开或者关闭断路器
  return metrics.getHealthCountsStream()
    .observe()
    .subscribe(new Subscriber() {
      @Override
      public void onCompleted() {

      }

      @Override
      public void onError(Throwable e) {

      }

      @Override
      public void onNext(HealthCounts hc) {
        //首先校验的时在时间窗范围内的请求次数，如果低于阈值（默认是20），不做处理，如果高于阈值，则去判断接口请求的错误率
        if (hc.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {           // 如果没有超过统计阈值的最低窗口值，就没有必要去改变断路器的状态
          // 当前如果断路器是关闭的，那么就保持关闭状态无需更改;
          // 如果断路器状态为半开状态，需要等待直到有成功的命令执行；
          // 如果断路器是打开状态,需要等待休眠窗口过期。
        } else {
          //判断接口请求的错误率（阈值默认是50），如果高于这个值，则断路器打开
          if (hc.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
    
            // 如果当前请求的错误率小于断路器设置的容错率百分比，也不会拦截请求
          } else {
            // 如果当前错误率太高则打开断路器
            if (status.compareAndSet(Status.CLOSED, Status.OPEN)) {
              circuitOpened.set(System.currentTimeMillis());
            }
          }
        }
      }
    });
}

每当metric 收到HealthCounts信息时就会调用next方法判断当前是否需要打开断路器。

HystrixCommandMetrics 是断路器最核心的东西，通过时间窗的形式，记录一段时间范围内（默认是10秒）的接口请求的健康状况（Command的执行状态，包括成功、失败、超时、线程池拒绝等）并得到对应的度量指标。

HealthCounts中保存了当前时间窗口内接口的请求状态包括请求总数、失败数和失败率，如果当前的请求总数和 失败率都达到阈值，则对接口进行熔断将断路器状态设置为打开状态记录当前打开断路器的时间。

失败回退

首先从HystrixCommand的queue()方法作为入口：

public Future queue() {
        final Future delegate = toObservable().toBlocking().toFuture();
  
  ......
  ......
  ......
    
}

queue()异步非堵塞的，它调用了toObservable().toBlocking().toFuture()方法，queue()执行完后，会创建一个新线程运行run()。

继续走到 toObservable() 方法：

public Observable toObservable() {
  
  ......
  ......
  ......
    
  final Func0> applyHystrixSemantics = new Func0>() {
    @Override
    public Observable call() {
      if (commandState.get().equals(CommandState.UNSUBSCRIBED)) {
        return Observable.never();
      }
      return applyHystrixSemantics(_cmd);
    }
  };
  
  ......
  ......
  ......
}

applyHystrixSemantics()封装Hystrix的核心流程，根据 HystrixCircuitBreaker 提供熔断器状态，确定执行run() 还是getFallback()：

private Observable applyHystrixSemantics(final AbstractCommand _cmd) {
  //如果回调逻辑抛出异常，那么就不会执行回调逻辑而是执行快速失败机制
  executionHook.onStart(_cmd);

  /* 断每个Hystrix命令的请求都通过它是否被执行 */
  if (circuitBreaker.attemptExecution()) {
    final TryableSemaphore executionSemaphore = getExecutionSemaphore();
    final AtomicBoolean semaphoreHasBeenReleased = new AtomicBoolean(false);
    final Action0 singleSemaphoreRelease = new Action0() {
      @Override
      public void call() {
        if (semaphoreHasBeenReleased.compareAndSet(false, true)) {
          executionSemaphore.release();
        }
      }
    };

    final Action1 markExceptionThrown = new Action1() {
      @Override
      public void call(Throwable t) {
        eventNotifier.markEvent(HystrixEventType.EXCEPTION_THROWN, commandKey);
      }
    };

    if (executionSemaphore.tryAcquire()) {
      try {
        /* 设置开始时间，用于监控执行超时*/
        executionResult = executionResult.setInvocationStartTime(System.currentTimeMillis());
        //包装HystrixCommand Observable，注册观察者
        return executeCommandAndObserve(_cmd)
          .doOnError(markExceptionThrown)
          .doOnTerminate(singleSemaphoreRelease)
          .doOnUnsubscribe(singleSemaphoreRelease);
      } catch (RuntimeException e) {
        return Observable.error(e);
      }
    } else {
      //信号量资源不足，拒绝执行，执行getFallBack()方法
      return handleSemaphoreRejectionViaFallback();
    }
  } else {
   //不可执行，快速失效，执行getFallBack()方法  
    return handleShortCircuitViaFallback();
  }
}

在包装executeCommandAndObserve的逻辑中有关于发生异常的时候对各种类型异常的判断，然后在doOnError()回调的时候对异常进行相应的处理。

private Observable executeCommandAndObserve(final AbstractCommand _cmd) {
  final HystrixRequestContext currentRequestContext = HystrixRequestContext.getContextForCurrentThread();

  ......

  final Func1> handleFallback = new Func1>() {
    @Override
    public Observable call(Throwable t) {
      circuitBreaker.markNonSuccess();
      //如果是异常情况，判断当前异常的类型
      Exception e = getExceptionFromThrowable(t);
      executionResult = executionResult.setExecutionException(e);
      //线程提交拒绝异常
      if (e instanceof RejectedExecutionException) {
        return handleThreadPoolRejectionViaFallback(e);
        //执行命令超时异常
      } else if (t instanceof HystrixTimeoutException) {
        return handleTimeoutViaFallback();
        //请求异常
      } else if (t instanceof HystrixBadRequestException) {
        return handleBadRequestByEmittingError(e);
      } else {
       //hystrix自定义异常
        if (e instanceof HystrixBadRequestException) {
          eventNotifier.markEvent(HystrixEventType.BAD_REQUEST, commandKey);
          return Observable.error(e);
        }

        return handleFailureViaFallback(e);
      }
    }
  };

  ......
  ......
  
  Observable execution;
  if (properties.executionTimeoutEnabled().get()) {
    execution = executeCommandWithSpecifiedIsolation(_cmd)
      .lift(new HystrixObservableTimeoutOperator(_cmd));
  } else {
    execution = executeCommandWithSpecifiedIsolation(_cmd);
  }

  return execution.doOnNext(markEmits)
    .doOnCompleted(markOnCompleted)
    .onErrorResumeNext(handleFallback)
    .doOnEach(setRequestContext);
}

注意到最后execution执行的是:executeCommandWithSpecifiedIsolation(_cmd)，根据隔离级别选择是“线程方式”隔离执行还是“信号量方式”隔离执行：

private Observable executeCommandWithSpecifiedIsolation(final AbstractCommand _cmd) {
  //如果当前配置的是线程池隔离的策略
  if (properties.executionIsolationStrategy().get() == ExecutionIsolationStrategy.THREAD) {
    
    return Observable.defer(new Func0>() {
      @Override
      public Observable call() {
        executionResult = executionResult.setExecutionOccurred();
        if (!commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.USER_CODE_EXECUTED)) {
          return Observable.error(new IllegalStateException("execution attempted while in state : " + commandState.get().name()));
        }

        metrics.markCommandStart(commandKey, threadPoolKey, ExecutionIsolationStrategy.THREAD);

        if (isCommandTimedOut.get() == TimedOutStatus.TIMED_OUT) {
          // the command timed out in the wrapping thread so we will return immediately
          // and not increment any of the counters below or other such logic
          return Observable.error(new RuntimeException("timed out before executing run()"));
        }
        if (threadState.compareAndSet(ThreadState.NOT_USING_THREAD, ThreadState.STARTED)) {
          //we have not been unsubscribed, so should proceed
          HystrixCounters.incrementGlobalConcurrentThreads();
          threadPool.markThreadExecution();
          // store the command that is being run
          endCurrentThreadExecutingCommand = Hystrix.startCurrentThreadExecutingCommand(getCommandKey());
          executionResult = executionResult.setExecutedInThread();
                //这里注册线程执行的hook，开启异步执行
          try {
            executionHook.onThreadStart(_cmd);
            executionHook.onRunStart(_cmd);
            executionHook.onExecutionStart(_cmd);
            return getUserExecutionObservable(_cmd);
          } catch (Throwable ex) {
            return Observable.error(ex);
          }
        } else {
          //command has already been unsubscribed, so return immediately
          return Observable.error(new RuntimeException("unsubscribed before executing run()"));
        }
      }
    }).doOnTerminate(new Action0() {
      @Override
      public void call() {
        if (threadState.compareAndSet(ThreadState.STARTED, ThreadState.TERMINAL)) {
          handleThreadEnd(_cmd);
        }
        if (threadState.compareAndSet(ThreadState.NOT_USING_THREAD, ThreadState.TERMINAL)) {
          //if it was never started and received terminal, then no need to clean up (I don't think this is possible)
        }
        //if it was unsubscribed, then other cleanup handled it
      }
    }).doOnUnsubscribe(new Action0() {
      @Override
      public void call() {
        if (threadState.compareAndSet(ThreadState.STARTED, ThreadState.UNSUBSCRIBED)) {
          handleThreadEnd(_cmd);
        }
        if (threadState.compareAndSet(ThreadState.NOT_USING_THREAD, ThreadState.UNSUBSCRIBED)) {
          //if it was never started and was cancelled, then no need to clean up
        }
        //if it was terminal, then other cleanup handled it
      }
      
    }).subscribeOn(threadPool.getScheduler(new Func0() {
      @Override
      public Boolean call() {
        return properties.executionIsolationThreadInterruptOnTimeout().get() && _cmd.isCommandTimedOut.get() == TimedOutStatus.TIMED_OUT;
      }
    }));
    //这里使用信号量机制
  } else {
    return Observable.defer(new Func0>() {
      @Override
      public Observable call() {
        executionResult = executionResult.setExecutionOccurred();
        if (!commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.USER_CODE_EXECUTED)) {
          return Observable.error(new IllegalStateException("execution attempted while in state : " + commandState.get().name()));
        }

        metrics.markCommandStart(commandKey, threadPoolKey, ExecutionIsolationStrategy.SEMAPHORE);
        // semaphore isolated
        // store the command that is being run
        endCurrentThreadExecutingCommand = Hystrix.startCurrentThreadExecutingCommand(getCommandKey());
        try {
          executionHook.onRunStart(_cmd);
          executionHook.onExecutionStart(_cmd);
          return getUserExecutionObservable(_cmd);  //the getUserExecutionObservable method already wraps sync exceptions, so this shouldn't throw
        } catch (Throwable ex) {
          //If the above hooks throw, then use that as the result of the run method
          return Observable.error(ex);
        }
      }
    });
  }
}