阿里限流神器sentinel与SpringBoot整合,实现根据request信息进行限流
阿里 sentinel 已适配 springboot，整合非常简单，只需要添加一些配置就可以了，不需要写任何代码。但是如果有一些特殊需求，需要更细粒度的控制限流，就需要自己写些代码，比如对外的接口需要根据请求者 ip，header 中的 user-agent 或者其他信息限流，来防止恶意刷接口，或者爬虫，本人的项目就遇到了有人使用 sqlmap 工具来寻找 sql 注入漏洞和爬虫扫描接口，于是研究了一下 sentinel，总结一下使用经验。
“主要的原理就是，根据 request 中的信息设置请求来源，然后使用正则表达式来匹配来源限流，比如把来源设置成 ip+user-agent，然后在限流规则的来源项中添加想要限流的正则表达式。
”
sentinel 与 springboot 整合没有设置请求来源，所以需要根据自己需求设置，sentinel 与来源的匹配只是使用简单的 equals 比较，这里使用正则表达式匹配更加灵活。
sentinel 项目地址：https://github.com/alibaba/Sentinel
sentinel-dashboard 控制台下载地址：https://github.com/alibaba/Sentinel/releases
sentinel-dashboard 控制台使用：https://github.com/alibaba/Sentinel/tree/master/sentinel-dashboard
先按照文档启动控制台
1. 与 springboot 整合
在 pom.xml 中添加依赖：
org.springframework.cloud
spring-cloud-alibaba-dependencies
0.2.1.RELEASE
pom
import
org.springframework.cloud
spring-cloud-starter-alibaba-sentinel
在 application.yml 中添加 sentinel 相关配置：
spring:
application:
# 项目名称，sentinel-dashboard中会显示
name: sentinel-example
cloud:
sentinel:
transport:
# 项目本地会启动一个服务的端口号，默认8719，用于与sentinel-dashboard通讯
port: 8719
# sentinel-dashboard服务地址
dashboard: localhost:9090
filter:
# 需要进行限流监控的接口，多个匹配用逗号隔开
url-patterns: /sentinel/*
servlet:
# 触发限流后重定向的页面
block-page: /sentinel/block
配置好后，启动项目，这时候控制台还看不到东西，需要请求一下接口就能看到请求接口统计的情况，至此就可以使用 sentinel 的各种功能。
与 springboot 的整合实际上是使用了一个过滤器 CommonFilter 拦截所有请求，但是在获取 entry 的过程中没有传入参数，所以不能使用热点参数这个功能。
2. 设置请求来源，使用正则表达式匹配来源限流
“只需要添加一个RequestOriginParser Bean，会自动设置到WebCellBackManager中。
”
@Configuration
public class SentinelConfig {
/**
* sentinel来源解析器
* @return
*/
@Bean
public RequestOriginParser requestOriginParser() {
return (request -> {
String remoteAddr = RequestUtil.getIpAddr(request);
String userAgent = request.getHeader("user-agent");
return String.join("|",
(remoteAddr == null ? "" : remoteAddr),
(userAgent == null ? "" : userAgent));
});
}
}
CommonFilter 在获取 entry 之前会使用 RequestOriginParser 从 request 中解析来源。
修改代码将来源匹配规则改成正则表达式匹配
需要重写 FlowSlot 和 FlowRuleChecker 两个类。虽然这两个类中各种 final 和私有方法，本意应该是不让改动这两个类的，但是通过研究源码，还是找到了办法，代码如下
新建类 RegexOriginFlowSlot 继承自 FlowSlot
public class RegexOriginFlowSlot extends FlowSlot {
@Override
public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count,
boolean prioritized, Object... args) throws Throwable {
checkFlow(resourceWrapper, context, node, count, prioritized);
fireEntry(context, resourceWrapper, node, count, prioritized, args);
}
void checkFlow(ResourceWrapper resource, Context context, DefaultNode node, int count, boolean prioritized) throws BlockException {
// Flow rule map cannot be null.
Map> flowRules = FlowRuleUtil.buildFlowRuleMap(FlowRuleManager.getRules());
List rules = flowRules.get(resource.getName());
if (rules != null) {
for (FlowRule rule : rules) {
if (!canPassCheck(rule, context, node, count, prioritized)) {
throw new FlowException(rule.getLimitApp());
}
}
}
}
boolean canPassCheck(FlowRule rule, Context context, DefaultNode node, int count, boolean prioritized) {
// 这里使用自己修改过的FlowRuleChecker，自定义来源匹配
return FlowRuleChecker.passCheck(rule, context, node, count, prioritized);
}
新建类 FlowRuleChecker，或者拷贝原来的类，只修改了 selectNodeByRequesterAndStrategy() 方法注释下面的代码，改成正则匹配
/**
* 拷贝原FlowRuleChecker代码，把来源检查改成正则匹配
*/
final class FlowRuleChecker {
static boolean passCheck(/*@NonNull*/ FlowRule rule, Context context, DefaultNode node, int acquireCount) {
return passCheck(rule, context, node, acquireCount, false);
}
static boolean passCheck(/*@NonNull*/ FlowRule rule, Context context, DefaultNode node, int acquireCount,
boolean prioritized) {
String limitApp = rule.getLimitApp();
if (limitApp == null) {
return true;
}
if (rule.isClusterMode()) {
return passClusterCheck(rule, context, node, acquireCount, prioritized);
}
return passLocalCheck(rule, context, node, acquireCount, prioritized);
}
private static boolean passLocalCheck(FlowRule rule, Context context, DefaultNode node, int acquireCount,
boolean prioritized) {
Node selectedNode = selectNodeByRequesterAndStrategy(rule, context, node);
if (selectedNode == null) {
return true;
}
return generateRater(rule).canPass(selectedNode, acquireCount);
}
private static TrafficShapingController generateRater(/*@Valid*/ FlowRule rule) {
if (rule.getGrade() == RuleConstant.FLOW_GRADE_QPS) {
switch (rule.getControlBehavior()) {
case RuleConstant.CONTROL_BEHAVIOR_WARM_UP:
return new WarmUpController(rule.getCount(), rule.getWarmUpPeriodSec(), ColdFactorProperty.coldFactor);
case RuleConstant.CONTROL_BEHAVIOR_RATE_LIMITER:
return new RateLimiterController(rule.getMaxQueueingTimeMs(), rule.getCount());
case RuleConstant.CONTROL_BEHAVIOR_WARM_UP_RATE_LIMITER:
return new WarmUpRateLimiterController(rule.getCount(), rule.getWarmUpPeriodSec(),
rule.getMaxQueueingTimeMs(), ColdFactorProperty.coldFactor);
case RuleConstant.CONTROL_BEHAVIOR_DEFAULT:
default:
// Default mode or unknown mode: default traffic shaping controller (fast-reject).
}
}
return new DefaultController(rule.getCount(), rule.getGrade());
}
static Node selectReferenceNode(FlowRule rule, Context context, DefaultNode node) {
String refResource = rule.getRefResource();
int strategy = rule.getStrategy();
if (StringUtil.isEmpty(refResource)) {
return null;
}
if (strategy == RuleConstant.STRATEGY_RELATE) {
return ClusterBuilderSlot.getClusterNode(refResource);
}
if (strategy == RuleConstant.STRATEGY_CHAIN) {
if (!refResource.equals(context.getName())) {
return null;
}
return node;
}
// No node.
return null;
}
private static boolean filterOrigin(String origin) {
// Origin cannot be `default` or `other`.
return !RuleConstant.LIMIT_APP_DEFAULT.equals(origin) && !RuleConstant.LIMIT_APP_OTHER.equals(origin);
}
static Node selectNodeByRequesterAndStrategy(/*@NonNull*/ FlowRule rule, Context context, DefaultNode node) {
// The limit app should not be empty.
String limitApp = rule.getLimitApp();
int strategy = rule.getStrategy();
String origin = context.getOrigin();
// 把来源改成正则匹配，只改了这里的代码
if (Pattern.compile(limitApp).matcher(origin).find() && filterOrigin(origin)) {
if (strategy == RuleConstant.STRATEGY_DIRECT) {
// Matches limit origin, return origin statistic node.
return context.getOriginNode();
}
return selectReferenceNode(rule, context, node);
} else if (RuleConstant.LIMIT_APP_DEFAULT.equals(limitApp)) {
if (strategy == RuleConstant.STRATEGY_DIRECT) {
// Return the cluster node.
return node.getClusterNode();
}
return selectReferenceNode(rule, context, node);
} else if (RuleConstant.LIMIT_APP_OTHER.equals(limitApp)
&& FlowRuleManager.isOtherOrigin(origin, rule.getResource())) {
if (strategy == RuleConstant.STRATEGY_DIRECT) {
return context.getOriginNode();
}
return selectReferenceNode(rule, context, node);
}
return null;
}
private static boolean passClusterCheck(FlowRule rule, Context context, DefaultNode node, int acquireCount,
boolean prioritized) {
try {
TokenService clusterService = pickClusterService();
if (clusterService == null) {
return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized);
}
long flowId = rule.getClusterConfig().getFlowId();
TokenResult result = clusterService.requestToken(flowId, acquireCount, prioritized);
return applyTokenResult(result, rule, context, node, acquireCount, prioritized);
// If client is absent, then fallback to local mode.
} catch (Throwable ex) {
RecordLog.warn("[FlowRuleChecker] Request cluster token unexpected failed", ex);
}
// Fallback to local flow control when token client or server for this rule is not available.
// If fallback is not enabled, then directly pass.
return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized);
}
private static boolean fallbackToLocalOrPass(FlowRule rule, Context context, DefaultNode node, int acquireCount,
boolean prioritized) {
if (rule.getClusterConfig().isFallbackToLocalWhenFail()) {
return passLocalCheck(rule, context, node, acquireCount, prioritized);
} else {
// The rule won't be activated, just pass.
return true;
}
}
private static TokenService pickClusterService() {
if (ClusterStateManager.isClient()) {
return TokenClientProvider.getClient();
}
if (ClusterStateManager.isServer()) {
return EmbeddedClusterTokenServerProvider.getServer();
}
return null;
}
private static boolean applyTokenResult(/*@NonNull*/ TokenResult result, FlowRule rule, Context context, DefaultNode node,
int acquireCount, boolean prioritized) {
switch (result.getStatus()) {
case TokenResultStatus.OK:
return true;
case TokenResultStatus.SHOULD_WAIT:
// Wait for next tick.
try {
Thread.sleep(result.getWaitInMs());
} catch (InterruptedException e) {
e.printStackTrace();
}
return true;
case TokenResultStatus.NO_RULE_EXISTS:
case TokenResultStatus.BAD_REQUEST:
case TokenResultStatus.FAIL:
return fallbackToLocalOrPass(rule, context, node, acquireCount, prioritized);
case TokenResultStatus.BLOCKED:
default:
return false;
}
}
private FlowRuleChecker() {}
static class ColdFactorProperty {
public static volatile int coldFactor = 3;
static {
String strConfig = SentinelConfig.getConfig(SentinelConfig.COLD_FACTOR);
if (StringUtil.isBlank(strConfig)) {
coldFactor = 3;
} else {
try {
coldFactor = Integer.valueOf(strConfig);
} catch (NumberFormatException e) {
RecordLog.info(e.getMessage(), e);
}
if (coldFactor <= 1) {
coldFactor = 3;
RecordLog.info("cold factor should be larger than 3");
}
}
}
}
}
类 FlowSlot 对应流量控制规则，调用流程如图
最终在 FlowRuleChecker.selectNodeByRequesterAndStrategy() 方法里匹配来源。
自定义 SlotChainBuilder，把 FlowSlot 替换为 RegexOriginFlowSlot
这样 RegexOriginFlowSlot 才能起作用
public class CustomSlotChainBuilder implements SlotChainBuilder {
@Override
public ProcessorSlotChain build() {
ProcessorSlotChain chain = new DefaultProcessorSlotChain();
chain.addLast(new NodeSelectorSlot());
chain.addLast(new ClusterBuilderSlot());
chain.addLast(new LogSlot());
chain.addLast(new StatisticSlot());
chain.addLast(new ParamFlowSlot());
chain.addLast(new SystemSlot());
chain.addLast(new AuthoritySlot());
// 把原来的FlowSlot替换成RegexOriginFlowSlot()
chain.addLast(new RegexOriginFlowSlot());
chain.addLast(new DegradeSlot());
return chain;
}
}
注册 CustomSlotChainBuilder
在项目的 resources 目录下新建文件: resources/META-INF/services/com.alibaba.csp.sentinel.slotchain.SlotChainBuilder，文件内容为 CustomSlotChainBuilder 的全类路径
“com.example.springboot.web.sentinel.CustomSlotChainBuilder
”
sentinel 使用的是 jdk 的 ServiceLoader 方式加载 SlotChainBuilder 的，实际只会加载除 DefaultSlotChainBuilder 之外的第一个 SlotChainBuilder，sentinel 本身还有一个 HotParamSlotChainBuilder 实现，所以最终是用到我们自己定义的 CustomSlotChainBuilder 还是 HotParamSlotChainBuilder 要看实际情况，最好在 CustomSlotChainBuilder 初始化时加点日志，看看有没有使用 CustomSlotChainBuilder，必要的时候可以在 pom.xml 中把 HotParamSlotChainBuilder 排除，HotParamSlotChainBuilder 所在的包是 sentinel-parameter-flow-control。
这样 CustomSlotChainBuilder 就会起作用。
“com.alibaba.csp.sentinel.slotchain.SlotChainProvider是加载SlotChainBuilder的类，下面是它加载SlotChainBuilder的源码
”
// jdk的ServiceLoader加载所有的SlotChainBuilder
private static final ServiceLoader LOADER = ServiceLoader.load(SlotChainBuilder.class);
private static void resolveSlotChainBuilder() {
List list = new ArrayList();
boolean hasOther = false;
for (SlotChainBuilder builder : LOADER) {
if (builder.getClass() != DefaultSlotChainBuilder.class) {
hasOther = true;
list.add(builder);
}
}
if (hasOther) {
// 拿到所有的SlotChainBuilder后只取第一个
builder = list.get(0);
} else {
// No custom builder, using default.
// 否则使用默认的
builder = new DefaultSlotChainBuilder();
}
RecordLog.info("[SlotChainProvider] Global slot chain builder resolved: "
+ builder.getClass().getCanonicalName());
}
至此再启动项目，就可以在流量控制的来源下使用正则表达式匹配来源，本例中可以过控制特定的 ip 或者 user-agent。
授权规则中的来源也使用正则匹配
同上面的修改原理一样，这里就只贴下代码
RegexOriginAuthoritySlot
public class RegexOriginAuthoritySlot extends AuthoritySlot {
@Override
public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
throws Throwable {
checkBlackWhiteAuthority(resourceWrapper, context);
fireEntry(context, resourceWrapper, node, count, prioritized, args);
}
void checkBlackWhiteAuthority(ResourceWrapper resource, Context context) throws AuthorityException {
Map> authorityRules = loadAuthorityConf(AuthorityRuleManager.getRules());
if (authorityRules == null) {
return;
}
Set rules = authorityRules.get(resource.getName());
if (rules == null) {
return;
}
for (AuthorityRule rule : rules) {
if (!AuthorityRuleChecker.passCheck(rule, context)) {
throw new AuthorityException(context.getOrigin());
}
}
}
private Map> loadAuthorityConf(List list) {
Map> newRuleMap = new ConcurrentHashMap<>();
if (list == null || list.isEmpty()) {
return newRuleMap;
}
for (AuthorityRule rule : list) {
if (!isValidRule(rule)) {
RecordLog.warn("[AuthorityRuleManager] Ignoring invalid authority rule when loading new rules: " + rule);
continue;
}
if (StringUtil.isBlank(rule.getLimitApp())) {
rule.setLimitApp(RuleConstant.LIMIT_APP_DEFAULT);
}
String identity = rule.getResource();
Set ruleSet = newRuleMap.get(identity);
// putIfAbsent
if (ruleSet == null) {
ruleSet = new HashSet<>();
ruleSet.add(rule);
newRuleMap.put(identity, ruleSet);
} else {
// One resource should only have at most one authority rule, so just ignore redundant rules.
RecordLog.warn("[AuthorityRuleManager] Ignoring redundant rule: " + rule.toString());
}
}
return newRuleMap;
}
public static boolean isValidRule(AuthorityRule rule) {
return rule != null && !StringUtil.isBlank(rule.getResource())
&& rule.getStrategy() >= 0 && StringUtil.isNotBlank(rule.getLimitApp());
}
}
AuthorityRuleChecker
/**
* 拷贝原AuthorityRuleChecker代码，把来源检查改成正则匹配
*/
final class AuthorityRuleChecker {
static boolean passCheck(AuthorityRule rule, Context context) {
String requester = context.getOrigin();
// Empty origin or empty limitApp will pass.
if (StringUtil.isEmpty(requester) || StringUtil.isEmpty(rule.getLimitApp())) {
return true;
}
boolean contain = false;
String[] patterns = rule.getLimitApp().split(",");
for (String pattern : patterns) {
// 把来源改成正则匹配
if (Pattern.compile(pattern).matcher(requester).find()) {
contain = true;
break;
}
}
int strategy = rule.getStrategy();
if (strategy == RuleConstant.AUTHORITY_BLACK && contain) {
return false;
}
if (strategy == RuleConstant.AUTHORITY_WHITE && !contain) {
return false;
}
return true;
}
private AuthorityRuleChecker() {}
}
CustomSlotChainBuilder
public class CustomSlotChainBuilder implements SlotChainBuilder {
@Override
public ProcessorSlotChain build() {
ProcessorSlotChain chain = new DefaultProcessorSlotChain();
chain.addLast(new NodeSelectorSlot());
chain.addLast(new ClusterBuilderSlot());
chain.addLast(new LogSlot());
chain.addLast(new StatisticSlot());
chain.addLast(new ParamFlowSlot());
chain.addLast(new SystemSlot());
// 把原来的AuthoritySlot替换成RegexOriginAuthoritySlot()
chain.addLast(new RegexOriginAuthoritySlot());
// 把原来的FlowSlot替换成RegexOriginFlowSlot()
chain.addLast(new RegexOriginFlowSlot());
chain.addLast(new DegradeSlot());
return chain;
}
}
3. 处理路径中带参数的问题
路径中带参数，如 / sentinel/hello/{param} 这样的接口，sentinel 会把不同的 param 值当作不同的接口来处理，这样会有性能问题。
需要使用前面用到的 WebCallbackManager 设置 UrlCleaner，这个接口的参数仅仅是请求路径，要在这个接口中把 / sentinel/hello/test，/sentinel/hello/test2，/sentinel/hello/test3 这样带参数值的路径转换成 / sentinel/hello/{param}。
要解决这个问题需要借助于 springmvc，因为 spring context 中包含有 / sentinel/hello/{param} 这样的路径信息，在研究了 springmvc 查找 HandlerMapping 的代码后，找到了解决办法，代码如下：
@Configuration
public class SentinelConfig {
/**
* sentinel来源解析器
* @return
*/
@Bean
public RequestOriginParser requestOriginParser() {
return (request -> {
String remoteAddr = RequestUtil.getIpAddr(request);
String userAgent = request.getHeader("user-agent");
return String.join("|",
(remoteAddr == null ? "" : remoteAddr),
(userAgent == null ? "" : userAgent));
});
}
/**
* sentinel路径处理器
* @param requestMappingHandlerMapping 从RequestMappingHandlerMapping中获取springmvc定义的接口路径
* @return
*/
@Bean
public UrlCleaner urlCleaner(RequestMappingHandlerMapping requestMappingHandlerMapping) {
// 如果接口路径包含参数，如/sentinel/hello/{param}，则设置路径解析器，处理带参数的url
// 参照spring匹配路径的源码
// 获取spring context的所有requestmapping，包含所有请求接口
// 获取所有接口路径，@GetMapping等注解中的路径
Map handlerMethods = requestMappingHandlerMapping.getHandlerMethods();
// 筛选出包含参数的Mapping，避免与无参数的路径匹配
List filteredHandlerMethods = new ArrayList<>();
Pattern pattern = Pattern.compile("\\{\\w+\\}");
handlerMethods.forEach((key, value) -> {
Set patterns = key.getPatternsCondition().getPatterns();
for (String path : patterns) {
if (pattern.matcher(path).find()) {
filteredHandlerMethods.add(key);
break;
}
}
});
if (!CollectionUtils.isEmpty(filteredHandlerMethods)) {
// 只处理包含参数的路径
return (originUrl -> {
// 逐个匹配路径
for (RequestMappingInfo requestMappingInfo : filteredHandlerMethods) {
PatternsRequestCondition patternsCondition = requestMappingInfo.getPatternsCondition();
// 匹配路径
List matchingPatterns = patternsCondition.getMatchingPatterns(originUrl);
if (matchingPatterns.size() > 0) {
if (matchingPatterns.size() > 1) {
matchingPatterns.sort(new AntPathMatcher().getPatternComparator(originUrl));
}
// 带参数的路径/sentinel/hello/abc，返回如/sentinel/hello/{param}
originUrl = matchingPatterns.get(0);
break;
}
}
return originUrl;
});
} else {
// 没有路径带参数的接口则使用默认的
return new DefaultUrlCleaner();
}
}
}
作者：noooooooon
来源链接：
https://blog.csdn.net/codemaster_2071/article/details/89479749?utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromBaidu%7Edefault-6.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EBlogCommendFromBaidu%7Edefault-6.control
