项目需求:
项目数据存储在hbase中,每次查询数据都需将一些数据进行转换,对于大数据查询操作频繁连接数据库获取字典值,这会影响整个查询速度。
解决方案:
1、根据不同业务模块划分,项目中对于数据流处理(单独spark服务),通过redis缓存字典数据。
2、前段需要用到字典数据,将数据缓存到.net端。
3、web端java服务,考虑到redis还需要安装,运维维护不变,开发人员使用也不方便,并切需要定时操作数据库缓存到redis,所以在服务端自定义缓存处理,通过定时器间隔一定时间,将数据写入到缓存中。
具体实现如下:
项目结构如下
在pom.xml中引入依赖包
xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0modelVersion> <groupId>com.example.springbootgroupId> <artifactId>springboot-cache2artifactId> <version>0.0.1-SNAPSHOTversion> <packaging>jarpackaging> <name>springboot-cache2name> <description>Demo project for Spring Bootdescription> <parent> <groupId>org.springframework.bootgroupId> <artifactId>spring-boot-starter-parentartifactId> <version>1.5.12.RELEASEversion> <relativePath/> parent> <properties> <project.build.sourceEncoding>UTF-8project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8project.reporting.outputEncoding> <java.version>1.7java.version> properties> <dependencies> <dependency> <groupId>org.springframework.bootgroupId> <artifactId>spring-boot-starter-webartifactId> dependency> <dependency> <groupId>org.springframework.bootgroupId> <artifactId>spring-boot-starter-testartifactId> <scope>testscope> dependency> <dependency> <groupId>org.mybatis.spring.bootgroupId> <artifactId>mybatis-spring-boot-starterartifactId> <version>1.3.1version> dependency> <dependency> <groupId>mysqlgroupId> <artifactId>mysql-connector-javaartifactId> dependency> <dependency> <groupId>com.github.pagehelpergroupId> <artifactId>pagehelperartifactId> <version>4.1.0version> dependency> <dependency> <groupId>org.quartz-schedulergroupId> <artifactId>quartzartifactId> <version>2.2.3version> dependency> <dependency> <groupId>commons-beanutilsgroupId> <artifactId>commons-beanutilsartifactId> <version>1.8.3version> dependency> <dependency> <groupId>commons-langgroupId> <artifactId>commons-langartifactId> <version>2.6version> dependency> <dependency> <groupId>log4jgroupId> <artifactId>log4jartifactId> <version>1.2.17version> dependency> <dependency> <groupId>dom4jgroupId> <artifactId>dom4jartifactId> <version>1.6.1version> dependency> <dependency> <groupId>jaxengroupId> <artifactId>jaxenartifactId> <version>1.1.4version> dependency> dependencies> <build> <plugins> <plugin> <groupId>org.springframework.bootgroupId> <artifactId>spring-boot-maven-pluginartifactId> plugin> plugins> build> project>
在application.properties中添加数据连接
#server.port=8090 #标示使用的是mysql/oracle/sqlserver datasource.type=mysql #mysql数据连接 spring.datasource.url=jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC spring.datasource.driver-class-name=com.mysql.jdbc.Driver spring.datasource.username=root spring.datasource.password=root #spring.datasource.max-active=20 #spring.datasource.max-idle=8 #spring.datasource.min-idle=8 #spring.datasource.initial-size=20 #mybatis 配置 # 配置映射文件加载 mybatis.mapper-locations=classpath*:mapper/*.xml # 实体类通过别名使用 #mybatis.type-aliases-package= #springmvc视图 spring.mvc.view.prefix=/WEB-INF/ spring.mvc.view.suffix=.jsp #单个文件上传限制 spring.http.multipart.maxFileSize=10Mb #单次文件上传限制 spring.http.multipart.maxRequestSize=100Mb
constant类:
package com.example.springboot.cache.constant; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; public class Constants { private static final Log log = LogFactory.getLog(Constants.class); public static String CacheXml = "cacheManage.xml"; /** * 配置xml文件路劲 */ public static String cacheXmlPath = null; static { if (null == Constants.class.getClassLoader().getResource("/")) { cacheXmlPath = Constants.class.getClassLoader().getResource("").getPath() + CacheXml; } else { cacheXmlPath = Constants.class.getClassLoader().getResource("/").getPath() + CacheXml; } if (cacheXmlPath != null && !"".equals(cacheXmlPath)) { String os = System.getProperty("os.name"); if (os.toLowerCase().startsWith("win") && cacheXmlPath.startsWith("/")) { cacheXmlPath = cacheXmlPath.substring(1); } } log.info("cacheXmlPath " + cacheXmlPath); } }
创建dao操作数据库类
package com.example.springboot.cache.dao; import com.example.springboot.cache.entity.dto.CacheEntity; import org.apache.ibatis.annotations.Mapper; import java.util.List; import java.util.Map; /** * @desc 查询数据字典 * @Author wangsh * @date 2018/5/6 18:10 * @return */ @Mapper public interface CacheManagerMapper { public List
mapper配置文件
xml version="1.0" encoding="UTF-8" ?> mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" > <mapper namespace="com.example.springboot.cache.dao.CacheManagerMapper"> <select id="getDataFromDBSaveToTempCache" parameterType="com.example.springboot.cache.entity.dto.CacheEntity" resultType="java.util.Map"> /*SELECT ${columns} FROM ${dbName}.${table} as rs WITH(NOLOCK) WHERE 1=1*/ select * from test.sc_dict_sex where 1=1 <if test="null != conditions and '' != conditions"> AND ${conditions} if> select> mapper>
创建实体类
package com.example.springboot.cache.entity.dto; import java.util.ArrayList; import java.util.List; import java.util.Map; /** * @desc 缓存实体类 * @Author wangsh * @date 2018/5/6 18:11 * @return */ public class CacheEntity { //数据库名称 private String dbName; //实例名 private String dbo; //表名 private String table; //数据库类型 private String datasourceType; //列名称(多个以逗号分割) private String columns; //查询条件 private String conditions; private String key; // 是否缓存 private String fullCache; private String toMapField; private Map> cacheMapData; private List
service层业务处理类
package com.example.springboot.cache.service; import com.example.springboot.cache.entity.dto.CacheEntity; import java.util.concurrent.ConcurrentMap; /** * @desc 缓存处理类 * @Author wangsh * @date 2018/5/6 18:06 * @return */ public interface PullDataToCache { public void pullData(ConcurrentMapcache); public void refreshCacheData(ConcurrentMap cache); }
service实现类
package com.example.springboot.cache.service.impl; import com.example.springboot.cache.constant.Constants; import com.example.springboot.cache.dao.CacheManagerMapper; import com.example.springboot.cache.entity.dto.CacheEntity; import com.example.springboot.cache.service.PullDataToCache; import com.example.springboot.cache.util.XmlUtils; import org.apache.commons.lang.StringUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.dom4j.Document; import org.dom4j.DocumentException; import org.dom4j.Element; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Service; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.concurrent.ConcurrentMap; /** * @desc 缓存处理类 * @Author wangsh * @date 2018/5/6 18:07 * @return */ @Service public class PullDataToCacheImpl implements PullDataToCache { private static final Log log = LogFactory.getLog(PullDataToCacheImpl.class); @Value("${datasource.type}") private String datasourceType; @Autowired private CacheManagerMapper mapper; private static ListcacheTableInfoList = new ArrayList (); @Override public void pullData(ConcurrentMap cache) { log.info("缓存管理-->服务启动:添加缓存入口开启调用!"); try { // 解析xml文件,并将要缓存的表信息保存入cacheTableInfoList; parseCacheXml(); // 从数据库获取数据,并保存数据 getDataFromDBSaveToTempCache(); // 放入缓存中 SaveToCacheManager(cache); } catch (Exception e) { log.error("缓存管理-->服务启动:缓存添加出现异常", e); } log.info("缓存管理-->服务启动:添加缓存入口调用完成!"); } private void SaveToCacheManager(ConcurrentMap cache) { for (CacheEntity ce : cacheTableInfoList) { String key = ce.getKey(); cache.put(key, ce); } cacheTableInfoList.clear(); } /** * @param @param ce * @return void * @author cjy * @Description: 针对配置需要独立成一个map的缓存数据进行遍历,并保存到全局map中 * @date 2017-6-8 */ private void cacheDataToMap(CacheEntity ce, List > list) { // 配置并遍历所有需要单独成map的字段 String toMapFiled = ce.getToMapField(); // 判断需要把数据独立成map的配置是否为空 if (StringUtils.isEmpty(toMapFiled)) { log.info("缓存管理-->添加缓存:" + ce.getTable() + "表没有要独立成map的字段返回!"); return; } if (null == list || list.size() == 0) { log.info("缓存管理-->添加缓存:" + ce.getTable() + "表没有查询出有效的数据返回!"); return; } String[] fields = toMapFiled.split(",", -1); for (int i = 0; i < fields.length; i++) { String field = fields[i]; // 获取哪个字段需要独立成map String[] keys = field.split(":", -1); if (null == keys || keys.length != 3) { log.info("缓存管理-->添加缓存:把数据添加到缓存map中出错,原因:" + ce.getTable() + "表配置的独立缓存没有配置!"); continue; } String tmpKey = keys[0]; String key = keys[1]; String value = keys[2]; log.debug("缓存管理-->添加缓存:独立缓存" + ce.getTable() + "表设置了独立map缓存,其中取独立缓存的key:" + tmpKey + " 缓存是以" + key + "字段值做key " + value + "字段值做value!"); Map tmp = new HashMap (); // 遍历所有的数据,并把数据添加到临时map中 for (Map data : list) { tmp.put(data.get(key).toString(), data.get(value)); } // 改组字段遍历完成,把临时map添加到全局对象中 ce.getCacheMapData().put(tmpKey, tmp); } } private void getDataFromDBSaveToTempCache() { for (CacheEntity ce : cacheTableInfoList) { try { // 查询数据 List > list = mapper.getDataFromDBSaveToTempCache(ce); if (null == ce.getList()) { ce.setList(new ArrayList >()); } // 判断是否需要把全量数据添加到缓存中 if ("true".equals(ce.getFullCache())) { // 添加全量数据到缓存中 ce.getList().addAll(list); } // 把需要独立成map的字段遍历,并添加到临时map中 cacheDataToMap(ce, list); } catch (Exception e) { log.error("缓存管理-->添加缓存:查询数据库报错", e); } } } /** * 解析缓存的xml文件 */ private void parseCacheXml() { try { String filePath = Constants.cacheXmlPath; Document doc = XmlUtils.loadFileByPath(filePath); Element ele = XmlUtils.rootElement(doc); List fileNodes = ele.selectNodes("/cacheManager/cache"); for (Element e : fileNodes) { CacheEntity ce = new CacheEntity(); ce.setDbName(e.attributeValue("dbName")); ce.setDbo(e.attributeValue("dbo")); ce.setTable(e.attributeValue("table")); //生成缓存key // String key = this.getDatabaseKey(e.attributeValue("dbName"), e.attributeValue("dbo"), e.attributeValue("table")); // ce.setKey(key); ce.setKey(e.attributeValue("dbName") + "_" + e.attributeValue("table")); ce.setDatasourceType(datasourceType); String columns = StringUtils.isEmpty(e.attributeValue("columns")) ? "*" : e.attributeValue("columns"); ce.setColumns(columns); ce.setConditions(e.attributeValue("conditions")); ce.setToMapField(e.attributeValue("toMapField")); String fullCache = StringUtils.isEmpty(e.attributeValue("fullCache")) ? "true" : e .attributeValue("fullCache"); ce.setFullCache(fullCache); ce.setCacheMapData(new HashMap >()); cacheTableInfoList.add(ce); } } catch (DocumentException e) { e.printStackTrace(); } } /** * 生成缓存的key * * @param dbName * @param dbo * @param table * @return */ private String getDatabaseKey(String dbName, String dbo, String table) { String key = null; if (StringUtils.isNotBlank(datasourceType)) { if ("mysql".equals(datasourceType.toLowerCase())) { key = dbName + "_" + table; } else if ("oracle".equals(datasourceType.toLowerCase())) { key = dbName + "_" + table; } else if ("sqlserver".equals(datasourceType.toLowerCase())) { key = dbName + "_" + dbo + "_" + table; } else { key = dbName + "_" + table; } } else { //默认mysql key = dbName + "_" + table; } return key; } /** * 刷新缓存中的数据 */ @Override public void refreshCacheData(ConcurrentMap cache) { log.info("缓存管理-->刷新缓存:缓存刷新开启调用!"); try { // 解析xml文件,并将要缓存的表信息保存入cacheTableInfoList; parseCacheXml(); // 从数据库获取数据,并保存数据 getDataFromDBSaveToTempCache(); // 放入缓存中 SaveToCacheManager(cache); } catch (Exception e) { log.error("缓存管理-->刷新缓存:缓存刷新出现异常", e); } log.info("缓存管理-->刷新缓存:缓存刷新调用完成!"); } }
xml解析工具类
package com.example.springboot.cache.util; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.dom4j.*; import org.dom4j.io.SAXReader; import java.io.File; import java.util.List; public class XmlUtils { private static Log log = LogFactory.getLog(XmlUtils.class); private static Document doc; public static Document loadFileByContent(String fileContent) throws DocumentException { doc = DocumentHelper.parseText(fileContent); return doc; } public static Document loadFileByPath(String FilePath) throws DocumentException { File file = new File(FilePath); if (file.exists()) { SAXReader reader = new SAXReader(); doc = reader.read(file); // 读取一个xml的文件 } else { log.info(FilePath + " xml文件不存在!"); } return doc; } public static Document loadFile(File file) throws DocumentException { SAXReader reader = new SAXReader(); doc = reader.read(file); // 读取一个xml的文件 return doc; } public static List selectNodes(Node node, String tiaojian) { Listnodelist = node.selectNodes(tiaojian); return nodelist; } public static Element rootElement(Document doc) { Element ele = doc.getRootElement(); return ele; } public static Element element(Element m, String name) { Element ele = m.element(name); return ele; } public static List elements(Element m, String name) { List elist = m.elements(name); return elist; } public static List elements(Element m) { List elist = m.elements(); return elist; } public static String attributeValue(Element m, String attriName) { String val = m.attributeValue(attriName); return val; } public static List attributeValues(Element m) { List vals = m.attributes(); return vals; } public static void main(String[] args) { String path = "C:\\Users\\Administrator\\Desktop\\sdfsdf\\cacheManage.xml"; try { Document doc = XmlUtils.loadFileByPath(path); Element ele = XmlUtils.rootElement(doc); List fileNodes = ele.selectNodes("/cacheManager/cache"); // Element e = (Element) e.selectSingleNode("/cacheManager/cache"); for (Element e : fileNodes) { System.out.println(e.attributeValue("dbName")); System.out.println(e.attributeValue("dbo")); System.out.println(e.attributeValue("table")); } } catch (DocumentException e) { e.printStackTrace(); } } }
创建缓存定时任务类
package com.example.springboot.cache.job; import com.example.springboot.cache.entity.dto.CacheEntity; import com.example.springboot.cache.service.PullDataToCache; import org.apache.commons.lang.StringUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.CommandLineRunner; import org.springframework.core.env.ConfigurableEnvironment; import org.springframework.stereotype.Component; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.concurrent.*; @Component public class CacheManager implements CommandLineRunner { private static final Log log = LogFactory.getLog(CacheManager.class); private static ConcurrentMap创建controller测试缓存类CACHE = new ConcurrentHashMap (); // @Autowired // private ServletContext servletContext; @Autowired private ConfigurableEnvironment env; @Autowired private PullDataToCache pullDataToCache; //每5分钟执行一次(单位秒) //@Value("${cachemanager_refresh_period}") protected String PERIOD = "300"; // 定时任务启动之后间隔时长开始执行定时任务 private final int INITIAL_DELAY = 10; // 定时任务的执行周期(单位:毫秒) private final int PERIOD_TMP = 5 * 60; @Override public void run(String... args) throws Exception { log.info("缓存管理-->缓存初始化:服务启动!"); refreshCacheData(); log.info(CACHE.toString()); log.info("缓存管理-->缓存初始化:服务加载结束!"); } /** * @param * @return void * @author cjy * @Description: 定时刷新缓存数据 * @date 2017-6-2 */ private void refreshCacheData() { ScheduledExecutorService executors = Executors.newSingleThreadScheduledExecutor(); if (StringUtils.isEmpty(PERIOD)) { // 当数据库配置的为空时走默认5分钟 executors.scheduleAtFixedRate(new CacheDataRefresh(), INITIAL_DELAY, PERIOD_TMP, TimeUnit.SECONDS); } else { executors.scheduleAtFixedRate(new CacheDataRefresh(), INITIAL_DELAY, Integer.parseInt(PERIOD), TimeUnit.SECONDS); } } // 加载缓存 public void load() { pullDataToCache.pullData(CACHE); } // 重新加载缓存 public void reload() { CACHE.clear(); load(); } // 获取缓存 public ConcurrentMap getCache() { return CACHE; } // 获取某个缓存 public List > getCache(String key) { CacheEntity cacheEntity = CACHE.get(key); if (cacheEntity == null) { return new ArrayList >(); } else { return CACHE.get(key).getList(); } } // 获取某个表的map缓存 public Map getCache(String key, String field) { CacheEntity cacheEntity = CACHE.get(key); if (cacheEntity == null) { return new HashMap (); } else { if (null == CACHE.get(key).getCacheMapData().get(field)) { return new HashMap (); } return CACHE.get(key).getCacheMapData().get(field); } } /** * @param @param tableKey 修改的表名称(例如:WFBDCMain_dbo_WFBDC_tb_Machine, * 表示是WFBDCMain数据库dbo对象中的WFBDC_tb_Machine表) * @param @param key 单独成map的key * @param @param subKey 要修改的子类map的key * @param @param subValue 要修改的子类map的key对应的值 * @return void * @author cjy * @Description: 修改某个表中具体一个map中的值, 例如:cacheManager.setCacheData("WFBDCMain_dbo_ISIP_tb_BaseInfo", "cacheCcicBaseInfo", "sdf", "fdasdfffffffff"); * @date 2017-7-5 */ public void setCacheData(String tableKey, String key, String subKey, Object subValue) { CacheEntity cacheEntity = CACHE.get(tableKey); if (null != cacheEntity) { // 获取子类map是否为空 if (null != CACHE.get(tableKey).getCacheMapData().get(key)) { // 更新缓存中的map数据 CACHE.get(tableKey).getCacheMapData().get(key).put(subKey, subValue); } } } // 缓存添加数据 public void put(String key, Map o, CacheEntity ce) { List > list = CACHE.get(key).getList(); if (list != null) { list.add(o); } else { list = ce.getList(); list.add(o); CACHE.put(key, ce); } } // 缓存中删除数据 public void remove(String key, Object o) { List > list = CACHE.get(key).getList(); if (list != null) { for (Object to : list) { if (to.equals(o)) { list.remove(o); } else { log.info(o + " 缓存中不存在该对象!"); } } } } // 销毁缓存 public void destory() { CACHE.clear(); } /** * @author cjy * @Description: 刷新缓存定时处理类 * @date 2017-6-2 下午6:48:36 */ private class CacheDataRefresh implements Runnable { @Override public void run() { try { pullDataToCache.refreshCacheData(CACHE); log.info("刷新缓存数据 :" + CACHE.toString()); } catch (Exception e) { log.error("缓存管理-->刷新缓存:定时调用刷新服务出现异常", e); } } } }
package com.example.springboot.cache.controller; import com.example.springboot.cache.job.CacheManager; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.ResponseBody; import org.springframework.web.bind.annotation.RestController; import java.util.List; import java.util.Map; /** * @desc 测试缓存加载数据 * @Author wangsh * @date 2018/5/6 18:56 */ @RestController @RequestMapping("/cache") public class CacheController { @Autowired private CacheManager cacheManager; @ResponseBody @RequestMapping("/hello") public List服务启动类> hello() { List > list = cacheManager.getCache("test_SC_dict_Sex"); System.out.println("缓存数据: " + list.toString()); return list; } }
package com.example.springboot.cache; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.context.annotation.ComponentScan; @SpringBootApplication @ComponentScan("com.example.springboot.*") public class SpringbootCache2Application { public static void main(String[] args) { SpringApplication.run(SpringbootCache2Application.class, args); } }
启动服务测试
测试如下,定时器每5分钟刷新一次,刷新频率可以根据项目情况自行调整。
访问服务测试
浏览器访问服务 http://localhost:8080/cache/hello ,后台调用缓存处理类并返回数据如下。
解析xml错误
D:\Program Files\maven-repository\dom4j\dom4j\1.6.1\dom4j-1.6.1.jar;D:\Program Files\maven-repository\xml-apis\xml-apis\1.4.01\xml-apis-1.4.01.jar" com.example.springboot.cache.util.XmlUtils
Exception in thread "main" java.lang.NoClassDefFoundError: org/jaxen/JaxenException
at org.dom4j.DocumentFactory.createXPath(DocumentFactory.java:230)
at org.dom4j.tree.AbstractNode.createXPath(AbstractNode.java:207)
at org.dom4j.tree.AbstractNode.selectNodes(AbstractNode.java:164)
at com.example.springboot.cache.util.XmlUtils.main(XmlUtils.java:80)
Caused by: java.lang.ClassNotFoundException: org.jaxen.JaxenException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 4 more
Process finished with exit code 1
项目中引入的dom4j-1.6.1版本,在解析时没有找到依赖包jaxen,导致解析失败。
解决办法:在pom.xml中引入依赖包
<dependency> <groupId>jaxengroupId> <artifactId>jaxenartifactId> <version>1.1.4version> dependency>