一、项目介绍
-------------------------------------------------
1.App开发商
每个开发商可以有多个App产品
2.App软件
3.数据服务平台提供商
Umeng，
向App开发商提供服务。提供App使用情况的统计服务。
4.SDK
数据服务平台提供商提供给App开发商的软件包。
内置log上报程序。
5.用户
每个使用App的设备。
6.租户
购买了数据服务平台提供商服务的App开发商。
二、项目分析
------------------------------------------------------
1.用户
设备id,唯一性
2.新增用户
首次打开应用的用户。
卸载再安装不是新增
3.活跃用户
指定时间段内打开过app的用户即为活跃用户。多次打开算一次。
4.月活率
活跃用户 / 截止到当月累计用户总数。
5.沉默用户
两天时间没有启动过app的用户就算沉默用户。
6.版本分布
计算各版本的新增用户、活跃用户、启动次数。
7.本周回流用户
上周没启动，本周启动的用户
8.连续n周活跃用户
连续n周，每周至少启动一次。
9.忠诚用户
连续5周以上活跃用户
10.连续活跃用户
连续2周以上
11.近期流失用户
连续n(2<= n <= 4)周没有启动应用的用户。
12.留存用户
某段时间内的新增用户，在经过一段时间后，仍然使用app的用户。
13.用户新鲜度
每天启动app的新老用户比例
14.单次使用时长
每次启动使用的时间长度。
15.日使用时长
每天的使用累加值。
16.启动次数计算标准
两次之间<30s,算作一次启动.
三、日志组成
--------------------------------------------------
1.启动日志
2.页面访问日志
3.事件日志
4.用户使用日志
5.错误日志
四、开始项目 -- 初始化web日志收集程序
----------------------------------------------------
1.创建新项目UmengProject
2.创建公共模块app-analyze-common，存放用于跨模块间访问的类。添加maven依赖
a.创建日志类AppLogEntity
package com.test.app.common;
/**
* AppLog实体类
* 内部含有各种日志时间的集合。
*/
public class AppLogEntity {
private String appId; //应用唯一标识
private String tenantId; //租户唯一标识,企业用户
private String deviceId; //设备唯一标识
private String appVersion; //版本
private String appChannel; //渠道,安装时就在清单中制定了，appStore等。
private String appPlatform; //平台
private String osType; //操作系统
private String deviceStyle; //机型
private AppStartupLog[] appStartupLogs; //启动相关信息的数组
private AppPageLog[] appPageLogs; //页面跳转相关信息的数组
private AppEventLog[] appEventLogs; //事件相关信息的数组
private AppUsageLog[] appUsageLogs; //app使用情况相关信息的数组
private AppErrorLog[] appErrorLogs; //错误相关信息的数组
/** get / set **/
}
b.抽取共性，创建日志基本父类AppBaseLog
package com.test.app.common;
import java.io.Serializable;
/**
* AppBaseLog
*/
public class AppBaseLog implements Serializable {
private Long createdAtMs; //日志创建时间
private String appId; //应用唯一标识
private String tenantId; //租户唯一标识,企业用户
private String deviceId; //设备唯一标识
private String appVersion; //版本
private String appChannel; //渠道,安装时就在清单中制定了，appStore等。
private String appPlatform; //平台
private String osType; //操作系统
private String deviceStyle; //机型
/*省略get set*/
}
c.创建日志具体分类的相关类AppStartupLog等
===================================================
package com.test.app.common;
/**
* 启动日志
*/
public class AppStartupLog extends AppBaseLog {
private String country; //国家，终端不用上报，服务器自动填充该属性
private String province; //省份，终端不用上报，服务器自动填充该属性
private String ipAddress; //ip地址
private String network; //网络
private String carrier; //运营商
private String brand; //品牌
private String deviceStyle; //机型
private String screenSize; //分辨率
private String osType; //操作系统
//省略getset
}
====================================================================
package com.test.app.common;
/**
* 应用上报的app错误日志相关信息
*/
public class AppErrorLog extends AppBaseLog {
private static final long serialVersionUID = 1L;
private String errorBrief; //错误摘要
private String errorDetail; //错误详情
}
=======================================================================
package com.test.app.common;
import java.util.Map;
/**
* 应用上报的事件相关信息
*
*/
public class AppEventLog extends AppBaseLog {
private static final long serialVersionUID = 1L;
private String eventId; //事件唯一标识
private Long eventDurationSecs; //事件持续时长
private Map paramKeyValueMap; //参数名/值对
}
=========================================================================
package com.test.app.common;
/**
* 应用上报的页面相关信息
*
*/
public class AppPageLog extends AppBaseLog {
private static final long serialVersionUID = 1L;
/*
* 一次启动中的页面访问次数(应保证每次启动的所有页面日志在一次上报中，即最后一条上报的页面记录的nextPage为空)
*/
private int pageViewCntInSession = 0;
private String pageId; //页面id
private int visitIndex = 0; //访问顺序号，0为第一个页面
private String nextPage; //下一个访问页面，如为空则表示为退出应用的页面
private Long stayDurationSecs = (long) 0; //当前页面停留时长
}
=============================================================================
package com.test.app.common;
/**
* 应用上报的使用时长相关信息
*/
public class AppUsageLog extends AppBaseLog {
private static final long serialVersionUID = 1L;
private Long singleUseDurationSecs; //单次使用时长(秒数),指一次启动内应用在前台的持续时长
private Long singleUploadTraffic; //单次使用过程中的上传流量
private Long singleDownloadTraffic; //单次使用过程中的下载流量
}
d.创建util工具包com.test.app.util和属性拷贝工具类PropertiesUtil
package com.test.app.util;
import java.beans.BeanInfo;
import java.beans.IntrospectionException;
import java.beans.Introspector;
import java.beans.PropertyDescriptor;
import java.lang.reflect.Method;
/**
* 通过内省实现属性复制
*/
public class PropertiesUtil {
//复制属性
//要将AppLogEntity中的所有相关属性的值赋值给对用的具体log的属性值[因为具体log的属性值是空的]
public static void copyProperties(Object src, Object des)
{
try {
BeanInfo srcInfo = Introspector.getBeanInfo(src.getClass());
//属性描述符
PropertyDescriptor[] sarr = srcInfo.getPropertyDescriptors();
for(PropertyDescriptor p : sarr)
{
Method getter = p.getReadMethod();
Method setter = p.getWriteMethod();
//获取set方法描述符的name
String setName = setter.getName();
//获取set方法的参数类型
Class [] param = setter.getParameterTypes();
try {
//通过get方法描述符，获取src的属性值
Object value = getter.invoke(src);
//通过src set方法描述符找到des的set方法，给des的属性赋值
Method desSettrer = des.getClass().getMethod(setName, param);
desSettrer.invoke(des, value);
} catch (Exception e) {
//出现异常说明des中没有src的属性
continue;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
3.创建app日志收集web模块app-log-collect-web
a.添加maven依赖
4.0.0
com.test
app-logs-collect-web
1.0-SNAPSHOT
junit
junit
4.11
com.fasterxml.jackson.core
jackson-core
2.8.8
com.fasterxml.jackson.core
jackson-databind
2.8.3
com.maxmind.db
maxmind-db
1.0.0
org.springframework
spring-webmvc
4.3.5.RELEASE
javax.servlet
servlet-api
2.5
com.test
app-analyze-common
1.0-SNAPSHOT
com.alibaba
fastjson
1.2.24
b.添加app-analyze-common模块依赖，共享模块资源
Project-Structure --> dependencies --> 3.Module De...
别忘记打钩，不然引用不了
c.创建java包
com.test.applogs.collect.web.controller
d.编写WEB-INF/web.xml
controller
org.springframework.web.servlet.DispatcherServlet
controller
/
e.创建新文件WEB-INF/controller-servlet.xml
f.创建日志收集控制器
package com.test.applogs.collect.web.controller;
import com.alibaba.fastjson.JSONObject;
import com.test.app.util.PropertiesUtil;
import com.test.app.common.*;
import com.test.app.common.AppLogEntity;
import com.test.app.common.AppStartupLog;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import javax.servlet.http.HttpServletRequest;
/**
*/
@Controller()
@RequestMapping("/coll")
public class CollectLogController {
/**
* 启动日志收集
*/
@RequestMapping(value = "/index", method = RequestMethod.POST)
@ResponseBody
public AppLogEntity collect(@RequestBody AppLogEntity e, HttpServletRequest req) {
System.out.println("=============================");
//server时间
long myTime = System.currentTimeMillis() ;
//客户端时间
long clientTime = Long.parseLong(req.getHeader("clientTime"));
//时间校对
long diff = myTime - clientTime ;
//对e进行处理，将具体日志分类的属性值填充完毕
processLogs(e);
//修正日志时间
verifyTime(e,diff);
String json = JSONObject.toJSONString(e);
System.out.println(json);
return e;
}
/**
* 校对各个具体日志的创建时间（使用服务器时间差diff）
*/
private void verifyTime(AppLogEntity e, long diff)
{
//启动修正
//startuplog
for(AppBaseLog log : e.getAppStartupLogs()){
log.setCreatedAtMs(log.getCreatedAtMs() + diff );
}
for(AppBaseLog log : e.getAppUsageLogs()){
log.setCreatedAtMs(log.getCreatedAtMs() + diff );
}
for(AppBaseLog log : e.getAppPageLogs()){
log.setCreatedAtMs(log.getCreatedAtMs() + diff );
}
for(AppBaseLog log : e.getAppEventLogs()){
log.setCreatedAtMs(log.getCreatedAtMs() + diff );
}
for(AppBaseLog log : e.getAppErrorLogs()){
log.setCreatedAtMs(log.getCreatedAtMs() + diff );
}
}
/**
* 将Log的属性分类复制到各个具体的log中
*/
private void processLogs(AppLogEntity e){
for(AppStartupLog log : e.getAppStartupLogs()){
PropertiesUtil.copyProperties(e,log);
}
for(AppErrorLog log : e.getAppErrorLogs()){
PropertiesUtil.copyProperties(e,log);
}
for(AppEventLog log : e.getAppEventLogs()){
PropertiesUtil.copyProperties(e,log);
}
for(AppPageLog log : e.getAppPageLogs()){
PropertiesUtil.copyProperties(e,log);
}
for(AppUsageLog log : e.getAppUsageLogs()){
PropertiesUtil.copyProperties(e,log);
}
}
}
g.创建tomcat-server启动程序
Run --> Edit Configurations --> + --> tomcat server --> ...
Project Structure --> Artifacts --> Add All Maven Jars --> ...
[注意！]Project Structure --> Artifacts --> Add Common模块到out class --> 不然依赖的其他模块是加载不到的，同时添加maven依赖的支持
4.创建app-logs-phone模块，用于模拟手机生成日志
a.添加maven依赖[可以通过maven的手段，达到访问其他模块的内容，比如common模块]
4.0.0
com.test
app-logs-phone
1.0-SNAPSHOT
com.alibaba
fastjson
1.2.24
com.test
app-analyze-common
1.0-SNAPSHOT
b.创建海量日志生成模拟程序
package com.test.app.client;
import com.alibaba.fastjson.JSONObject;
import com.test.app.common.*;
//import com.test.app.util.PropertiesUtil;
import java.io.File;
import java.io.FileWriter;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import java.util.Random;
/**
* 数据生成程序
*/
public class TestGenData {
/**
*
*/
private static String url = "http://localhost:8080/coll/index";
private static Random random = new Random();
private static String appId = "sdk34734";
private static String[] tenantIds = {"cake"};
private static String[] deviceIds = initDeviceId();
private static String[] appVersions = {"3.2.1", "3.2.2"};
private static String[] appChannels = {"youmeng1", "youmeng2"};
private static String[] appPlatforms = {"android", "ios"};
private static Long[] createdAtMsS = initCreatedAtMs();
//国家，终端不用上报，服务器自动填充该属性
private static String[] countrys = {"America", "china"};
//省份，终端不用上报，服务器自动填充该属性
private static String[] provinces = {"Washington", "jiangxi", "beijing"};
//网络
private static String[] networks = {"WiFi", "CellNetwork"};
//运营商
private static String[] carriers = {"中国移动", "中国电信", "EE"};
//机型
private static String[] deviceStyles = {"iPhone 6", "iPhone 6 Plus", "红米手机1s"};
//分辨率
private static String[] screenSizes = {"1136*640", "960*640", "480*320"};
//操作系统
private static String[] osTypes = {"8.3", "7.1.1"};
//品牌
private static String[] brands = {"三星", "华为", "Apple", "魅族", "小米", "锤子"};
//事件唯一标识
private static String[] eventIds = {"popMenu", "autoImport", "BookStore"};
//事件持续时长
private static Long[] eventDurationSecsS = {new Long(25), new Long(67), new Long(45)};
static Map map1 = new HashMap() {
{
put("testparam1key", "testparam1value");
put("testparam2key", "testparam2value");
}
};
static Map map2 = new HashMap() {
{
put("testparam3key", "testparam3value");
put("testparam4key", "testparam4value");
}
};
private static Map[] paramKeyValueMapsS = {map1, map2};
//单次使用时长(秒数),指一次启动内应用在前台的持续时长
private static Long[] singleUseDurationSecsS = initSingleUseDurationSecs();
private static String[] errorBriefs = {"at cn.lift.dfdf.web.AbstractBaseController.validInbound(AbstractBaseController.java:72)", "at cn.lift.appIn.control.CommandUtil.getInfo(CommandUtil.java:67)"}; //错误摘要
private static String[] errorDetails = {"java.lang.NullPointerException\\n " + "at cn.lift.appIn.web.AbstractBaseController.validInbound(AbstractBaseController.java:72)\\n " + "at cn.lift.dfdf.web.AbstractBaseController.validInbound", "at cn.lift.dfdfdf.control.CommandUtil.getInfo(CommandUtil.java:67)\\n " + "at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n" + " at java.lang.reflect.Method.invoke(Method.java:606)\\n"}; //错误详情
//页面id
private static String[] pageIds = {"list.html", "main.html", "test.html"};
//访问顺序号，0为第一个页面
private static int[] visitIndexs = {0, 1, 2, 3, 4};
//下一个访问页面，如为空则表示为退出应用的页面
private static String[] nextPages = {"list.html", "main.html", "test.html", null};
//当前页面停留时长
private static Long[] stayDurationSecsS = {new Long(45), new Long(2), new Long(78)};
//启动相关信息的数组
private static AppStartupLog[] appStartupLogs = initAppStartupLogs();
//页面跳转相关信息的数组
private static AppPageLog[] appPageLogs = initAppPageLogs();
//事件相关信息的数组
private static AppEventLog[] appEventLogs = initAppEventLogs();
//app使用情况相关信息的数组
private static AppUsageLog[] appUsageLogs = initAppUsageLogs();
//错误相关信息的数组
private static AppErrorLog[] appErrorLogs = initAppErrorLogs();
private static String[] initDeviceId() {
String base = "device22";
String[] result = new String[100];
for (int i = 0; i < 100; i++) {
result[i] = base + i + "";
}
return result;
}
private static Long[] initCreatedAtMs() {
Long createdAtMs = System.currentTimeMillis();
Long[] result = new Long[11];
for (int i = 0; i < 10; i++) {
result[i] = createdAtMs - (long) (i * 24 * 3600 * 1000);
}
result[10] = createdAtMs;
return result;
}
private static Long[] initSingleUseDurationSecs() {
Random random = new Random();
Long[] result = new Long[200];
for (int i = 1; i < 200; i++) {
result[i] = (long) random.nextInt(200);
}
return result;
}
//启动相关信息的数组
private static AppStartupLog[] initAppStartupLogs() {
AppStartupLog[] result = new AppStartupLog[10];
for (int i = 0; i < 10; i++) {
AppStartupLog appStartupLog = new AppStartupLog();
appStartupLog.setCountry(countrys[random.nextInt(countrys.length)]);
appStartupLog.setProvince(provinces[random.nextInt(provinces.length)]);
appStartupLog.setNetwork(networks[random.nextInt(networks.length)]);
appStartupLog.setCarrier(carriers[random.nextInt(carriers.length)]);
appStartupLog.setDeviceStyle(deviceStyles[random.nextInt(deviceStyles.length)]);
appStartupLog.setScreenSize(screenSizes[random.nextInt(screenSizes.length)]);
appStartupLog.setOsType(osTypes[random.nextInt(osTypes.length)]);
appStartupLog.setBrand(brands[random.nextInt(brands.length)]);
appStartupLog.setCreatedAtMs(createdAtMsS[random.nextInt(createdAtMsS.length)]);
result[i] = appStartupLog;
}
return result;
}
//页面跳转相关信息的数组
private static AppPageLog[] initAppPageLogs() {
AppPageLog[] result = new AppPageLog[10];
for (int i = 0; i < 10; i++) {
AppPageLog appPageLog = new AppPageLog();
String pageId = pageIds[random.nextInt(pageIds.length)];
int visitIndex = visitIndexs[random.nextInt(visitIndexs.length)];
String nextPage = nextPages[random.nextInt(nextPages.length)];
while (pageId.equals(nextPage)) {
nextPage = nextPages[random.nextInt(nextPages.length)];
}
Long stayDurationSecs = stayDurationSecsS[random.nextInt(stayDurationSecsS.length)];
appPageLog.setPageId(pageId);
appPageLog.setStayDurationSecs(stayDurationSecs);
appPageLog.setVisitIndex(visitIndex);
appPageLog.setNextPage(nextPage);
appPageLog.setCreatedAtMs(createdAtMsS[random.nextInt(createdAtMsS.length)]);
result[i] = appPageLog;
}
return result;
}
;
//事件相关信息的数组
private static AppEventLog[] initAppEventLogs() {
AppEventLog[] result = new AppEventLog[10];
for (int i = 0; i < 10; i++) {
AppEventLog appEventLog = new AppEventLog();
appEventLog.setEventId(eventIds[random.nextInt(eventIds.length)]);
appEventLog.setParamKeyValueMap(paramKeyValueMapsS[random.nextInt(paramKeyValueMapsS.length)]);
appEventLog.setEventDurationSecs(eventDurationSecsS[random.nextInt(eventDurationSecsS.length)]);
appEventLog.setCreatedAtMs(createdAtMsS[random.nextInt(createdAtMsS.length)]);
result[i] = appEventLog;
}
return result;
}
;
//app使用情况相关信息的数组
private static AppUsageLog[] initAppUsageLogs() {
AppUsageLog[] result = new AppUsageLog[10];
for (int i = 0; i < 10; i++) {
AppUsageLog appUsageLog = new AppUsageLog();
appUsageLog.setSingleUseDurationSecs(singleUseDurationSecsS[random.nextInt(singleUseDurationSecsS.length)]);
appUsageLog.setCreatedAtMs(createdAtMsS[random.nextInt(createdAtMsS.length)]);
result[i] = appUsageLog;
}
return result;
}
;
//错误相关信息的数组
private static AppErrorLog[] initAppErrorLogs() {
AppErrorLog[] result = new AppErrorLog[10];
for (int i = 0; i < 10; i++) {
AppErrorLog appErrorLog = new AppErrorLog();
appErrorLog.setErrorBrief(errorBriefs[random.nextInt(errorBriefs.length)]);
appErrorLog.setErrorDetail(errorDetails[random.nextInt(errorDetails.length)]);
appErrorLog.setCreatedAtMs(createdAtMsS[random.nextInt(createdAtMsS.length)]);
appErrorLog.setOsType(osTypes[random.nextInt(osTypes.length)]);
appErrorLog.setDeviceStyle(deviceStyles[random.nextInt(deviceStyles.length)]);
result[i] = appErrorLog;
}
return result;
}
private static void httpPost(String urlString, String params) {
URL url;
try {
url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setUseCaches(false);
conn.setInstanceFollowRedirects(true);
conn.setRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0");
conn.setRequestProperty("Content-Type", "application/json");
conn.setConnectTimeout(1000 * 5);
conn.connect();
conn.getOutputStream().write(params.getBytes("utf8"));
conn.getOutputStream().flush();
conn.getOutputStream().close();
byte[] buffer = new byte[1024];
StringBuffer sb = new StringBuffer();
InputStream in = conn.getInputStream();
int httpCode = conn.getResponseCode();
System.out.println(in.available());
while (in.read(buffer, 0, 1024) != -1) {
sb.append(new String(buffer));
}
System.out.println("sb:" + sb.toString());
in.close();
System.out.println(httpCode);
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
Test1();
}
private static void Test1() {
Random random = new Random();
try {
//发送数据
for (int i = 1; i <= 2000; i++) {
AppLogEntity logEntity = new AppLogEntity();
//渠道
logEntity.setAppChannel(appChannels[random.nextInt(appChannels.length)]);
//appid
logEntity.setAppId(appId);
//platform
logEntity.setAppPlatform(appPlatforms[random.nextInt(appPlatforms.length)]);
logEntity.setAppVersion(appVersions[random.nextInt(appVersions.length)]);
String tenantId = tenantIds[random.nextInt(tenantIds.length)];
if (tenantId != null) {
logEntity.setTenantId(tenantId);
}
logEntity.setTenantId(tenantIds[random.nextInt(tenantIds.length)]);
logEntity.setDeviceId(deviceIds[random.nextInt(deviceIds.length)]);
//模拟startup log集合
logEntity.setAppStartupLogs(new AppStartupLog[]{appStartupLogs[random.nextInt(appStartupLogs.length)]});
logEntity.setAppEventLogs(new AppEventLog[]{appEventLogs[random.nextInt(appEventLogs.length)]});
logEntity.setAppErrorLogs(new AppErrorLog[]{appErrorLogs[random.nextInt(appErrorLogs.length)]});
logEntity.setAppPageLogs(new AppPageLog[]{appPageLogs[random.nextInt(appPageLogs.length)]});
logEntity.setAppUsageLogs(new AppUsageLog[]{appUsageLogs[random.nextInt(appUsageLogs.length)]});
try {
//将对象转换成json string
String json = JSONObject.toJSONString(logEntity);
UploadUtil.upload(json);
Thread.sleep(2000);
} catch (Exception ex) {
System.out.println(ex);
}
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
private static void Test2() {
boolean result = map1.isEmpty();
System.out.println(result);
}
}
c.创建UploadUtil类，模拟日志上传到web
package com.test.app.client;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
/**
* 模拟手机上报日志程序
*/
public class UploadUtil {
/**
* 上传日志
*/
public static void upload(String json) throws Exception {
try{
//输入流
InputStream in = ClassLoader.getSystemResourceAsStream("log.json");
URL url = new URL("http://localhost:8080/coll/index");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
//设置请求方式为post
conn.setRequestMethod("POST");
//时间头用来供server进行时钟校对的
conn.setRequestProperty("clientTime",System.currentTimeMillis() + "");
//允许上传数据
conn.setDoOutput(true);
//设置请求的头信息,设置内容类型
conn.setRequestProperty("Content-Type", "application/json");
//输出流
OutputStream out = conn.getOutputStream();
out.write(json.getBytes());
out.flush();
out.close();
in.close();
int code = conn.getResponseCode();
System.out.println(code);
}
catch (Exception e){
e.printStackTrace();
}
}
}
d.生产环境下特别注意时钟问题
手机客户端的时间可能不准，所以不能按照客户端提供的时间来算
所以。服务器端收集文件之后要进行时间校对
1)client
发送数据同时，写入clientTime头。
conn.setRequestProperty("clientTime",System.currentTimeMillis() + "");
2)web server
//server时间
long myTime = System.currentTimeMillis() ;
//客户端时间
long clientTime = Long.parseLong(req.getHeader("clientTime"));
//时间校对
long diff = myTime - clientTime ;
3)完成Log实体中公共部分属性和Log类中间属性复制。
