Redis高级类型(统计全站访问量，日活跃用户)
-
HyperLogLog(超级日志)(统计访客)
-
- 采用一种基数算法，用于完成独立总数的统计
- 占据空间小，无论统计多少个数据，只占12K的内存空间
- 不精确的统计算法，标准误差为0.81%
-
Bitmap(位图)
-
- 不是一种独立的数据结构，实际上就是字符串
- 支持按位存取数据，可以将其看成是byte数组
- 适合存储大量的连续的数据的布尔值
两种数据结构都适合用来统计网站运营数据，节约内存。
HyperLogLog
测试使用HyperLogLog来统计200000次访问去重
/**
* 统计20万重复数据的独立总数
*/
@Test
public void testHyperLogLog(){
String redisKey = "test:h11:01";
for (int i = 1; i <= 100000; i++) {
redisTemplate.opsForHyperLogLog().add(redisKey,i);
}
for (int i = 1; i <= 100000; i++) {
redisTemplate.opsForHyperLogLog().add(redisKey,(int)Math.random()*100000+1);
}
System.out.println(redisTemplate.opsForHyperLogLog().size(redisKey));
}
结果：
虽然有误差但是完全可以接受
测试使用HyperLogLog来统计三组数据合并去重
//将三组数据合并，在合并后的重复数据中统计独立总数
@Test
public void testHyperLogLogUnion(){
String redisKey1 = "test:h11:02";
for (int i = 1; i <= 10000 ; i++) {
redisTemplate.opsForHyperLogLog().add(redisKey1,i);
}
String redisKey2 = "test:h11:02";
for (int i = 5001; i <= 15000 ; i++)
redisTemplate.opsForHyperLogLog().add(redisKey2,i);
String redisKey3 = "test:h11:02";
for (int i = 10001; i <= 20000 ; i++)
redisTemplate.opsForHyperLogLog().add(redisKey3,i);
String redisKey = "test:h11:union";
redisTemplate.opsForHyperLogLog().union(redisKey,redisKey1,redisKey2,redisKey3);
System.out.println( redisTemplate.opsForHyperLogLog().size(redisKey) );
}
将数据合并并且进行查重，结果
**
**
BitMap操作
可以用来统计签到数据
测试使用BitMap来统计数据
@Test
public void testBitMap(){
String redisKey = "test:bm:01";
//记录 签到
redisTemplate.opsForValue().setBit(redisKey,1,true);
redisTemplate.opsForValue().setBit(redisKey,4,true);
redisTemplate.opsForValue().setBit(redisKey,7,true);
//查询
System.out.println(redisTemplate.opsForValue().getBit(redisKey,0));
System.out.println(redisTemplate.opsForValue().getBit(redisKey,1));
System.out.println(redisTemplate.opsForValue().getBit(redisKey,2));
//统计
Object obj = redisTemplate.execute( (RedisCallback) connection ->{
return connection.bitCount(redisKey.getBytes());
});
System.out.println(obj);
}
需要使用redisCallback方法，传入key值的byte数组
测试使用BitMap进行OR运算
//统计3组数据的布尔值，并对这3组数据做OR运算
@Test
public void testBitMapOperation(){
String rediskey = "test:bm:02";
redisTemplate.opsForValue().setBit(rediskey,0,true);
redisTemplate.opsForValue().setBit(rediskey,1,true);
redisTemplate.opsForValue().setBit(rediskey,2,true);
String rediskey2 = "test:bm:03";
redisTemplate.opsForValue().setBit(rediskey2,2,true);
redisTemplate.opsForValue().setBit(rediskey2,3,true);
redisTemplate.opsForValue().setBit(rediskey2,4,true);
String rediskey3 = "test:bm:04";
redisTemplate.opsForValue().setBit(rediskey3,4,true);
redisTemplate.opsForValue().setBit(rediskey3,5,true);
redisTemplate.opsForValue().setBit(rediskey3,6,true);
String rediskey4 = "test:or";
Object obj = redisTemplate.execute( (RedisCallback) connection ->{
connection.bitOp(RedisStringCommands.BitOperation.OR,
rediskey4.getBytes(),rediskey.getBytes(),rediskey2.getBytes(),rediskey3.getBytes());
return connection.bitCount(rediskey4.getBytes());
});
System.out.println(obj);
System.out.println(redisTemplate.opsForValue().getBit(rediskey4,0));
System.out.println(redisTemplate.opsForValue().getBit(rediskey4,1));
System.out.println(redisTemplate.opsForValue().getBit(rediskey4,2));
System.out.println(redisTemplate.opsForValue().getBit(rediskey4,3));
System.out.println(redisTemplate.opsForValue().getBit(rediskey4,4));
System.out.println(redisTemplate.opsForValue().getBit(rediskey4,5));
System.out.println(redisTemplate.opsForValue().getBit(rediskey4,6));
}
结果
Redis高级数据类型实战
在项目中用作网站数据统计
-
UV(Unique Visitor)
-
- 独立访客，需通过用户IP排重统计数据
- 每次访问都要进行统计
- HyperLogLog，性能好，且存储空间小
-
DAU(Daily Active User)
-
- 日活跃用户，通过用户ID排重统计数据。
- 访问过一次，认为其活跃
- Bitmap，性能好、且可以统计精确的结果
UV实战
redis已经将基本操作封装完毕，所以我们直接编写service即可
/**
* 提供统计网站访客功能
* @author ：LY
* @date ：Created in 2021/3/11 22:20
* @modified By：
*/
@Service
public class DataService {
@Autowired
private RedisTemplate redisTemplate;
private SimpleDateFormat df = new SimpleDateFormat("yyyyMMdd");
/**
* 将指定IP计入UV
* @param ip
*/
public void recordUV(String ip){
String redisKey = RedisKeyUtil.getUVKey(df.format(new Date()));
redisTemplate.opsForHyperLogLog().add(redisKey,ip);
}
/**
* 统计置顶范围内的id数量
* @param start
* @param end
* @return
*/
public long calculateUV(Date start,Date end){
if (start == null || end == null){
throw new IllegalArgumentException("参数不能为空");
}
//整理该日期范围的Key
List<String> keyList = new ArrayList<>();
Calendar calendar = Calendar.getInstance();
calendar.setTime(start);
while (!calendar.getTime().after(end)){
String key = RedisKeyUtil.getUVKey(df.format(calendar.getTime()));
keyList.add(key);
calendar.add(Calendar.DATE,1);
}
//合并数据
String redisKey = RedisKeyUtil.getUVKey(df.format(start),df.format(end));
redisTemplate.opsForHyperLogLog().union(redisKey,keyList.toArray());
//返回统计结果
return redisTemplate.opsForHyperLogLog().size(redisKey);
}
/**
* 统计日活跃用户
* @param userId
*/
public void recordDAU(int userId){
String redisKey = RedisKeyUtil.getDAUKey(df.format(new Date()));
redisTemplate.opsForValue().setBit(redisKey,userId,true);
}
/**
* 统计指定日期范围的DAU数量
* @param start
* @param end
* @return
*/
public long calculateDAU(Date start,Date end){
if (start == null || end == null){
throw new IllegalArgumentException("参数不能为空");
}
//整理该日期范围的Key
List<byte[]> keyList = new ArrayList<>();
Calendar calendar = Calendar.getInstance();
calendar.setTime(start);
while (!calendar.getTime().after(end)){
String key = RedisKeyUtil.getDAUKey(df.format(calendar.getTime()));
keyList.add(key.getBytes());
calendar.add(Calendar.DATE,1);
}
//进行OR运算
return (long) redisTemplate.execute((RedisCallback) connection -> {
String redisKey = RedisKeyUtil.getDAUKey(df.format(start),df.format(end));
connection.bitOp(RedisStringCommands.BitOperation.OR,
redisKey.getBytes(),keyList.toArray(new byte[0][0]));//转成byte[0][0]格式
return connection.bitCount(redisKey.getBytes());
});
}
}
service中实现了UV和DAU的基本操作，以uv为例，编写了单个添加，与时间范围查询，单个添加用于在拦截器中统计访客，范围查询用来根据时间区间统计访问量数据，DAU基本同理。
接下来编写拦截器，每次用用户访问服务器获取服务时记录。
/**
* @author ：LY
* @date ：Created in 2021/3/11 22:40
* @modified By：
*/
@Component
public class DataInterceptor implements HandlerInterceptor {
@Autowired
private DataService dataService;
@Autowired
private HostHolder hostHolder;
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
//统计UV
String ip = request.getRemoteHost();
dataService.recordUV(ip);
//统计DAU
User user = hostHolder.getUser();
if (user!=null){
dataService.recordDAU(user.getId());
}
return true;
}
}
每个访客通过Ip来记录，dau通过获取用户id存储，如果未登录就不算活跃用户。
拦截器编写完毕后需要在WebMvcConfig来进行配置放行路径，静态资源不做拦截。
registry.addInterceptor(dataInterceptor)
.excludePathPatterns("/**/*.css","/**/*.js","/**/*.png","/**/*.jpg","/**/*.jpeg");
最后一步编写controller提供数据获取接口
/**
* @author ：LY
* @date ：Created in 2021/3/11 22:43
* @modified By：
*/
@Controller
public class DataController {
@Autowired
private DataService dataService;
@RequestMapping(path = "/data",method = {
RequestMethod.GET,RequestMethod.POST})
public String getDataPage(){
return "/site/admin/data";
}
@PostMapping("/data/uv")
public String getUV(@DateTimeFormat(pattern = "yyyy-MM-dd") Date start,
@DateTimeFormat(pattern = "yyyy-MM-dd")Date end,
Model model){
long uv = dataService.calculateUV(start,end);
model.addAttribute("uvResult",uv);
model.addAttribute("uvStartDate",start);
model.addAttribute("uvEndDate",end);
return "forward:/data";
}
@PostMapping("/data/dau")
public String getDAU(@DateTimeFormat(pattern = "yyyy-MM-dd") Date start,
@DateTimeFormat(pattern = "yyyy-MM-dd")Date end,
Model model){
long dau = dataService.calculateDAU(start,end);
model.addAttribute("dauResult",dau);
model.addAttribute("dauStartDate",start);
model.addAttribute("dauEndDate",end);
return "forward:/data";
}
}
最后来看看实际效果
