Qlik之数据如何做增量(三)

1.需求

   对于数据量很大的需求来说，例如几千万，上亿条数据，qlik要加载很长时间，是相当的费时间，所以这个记录一下，如何在qlik中做数据的增量

2.操作步骤

1.先跑一遍全量数据，存储在qlik服务器里(例如取20190101到20191210的数据，那我就先跑20190101到20191201的全量数据)

   

1.  select * from test where time >= '20190101' and where time < '20191201'  
2.  Store  用户数据  into [lib://test_qvd (win-i1nf3g039d9_qs_chanpin_dev)/用户数据201901_201912.qvd];   

    上面的第二句，就是要存储在qlik服务器中的路径，存入一个qvd文件(全量的数据)。

 

2. 然后在自动生成的脚本里面，加上全量和增量的代码，如下：

 

Unqualify *;

[用户数据_temp_da60bde4-1ebd-e1e9-2a3f-1f7a1a09]:
//增量数据
LOAD
 [a] as [b],
 [c] as [d]
RESIDENT [用户数据];
concatenate([用户数据_temp_da60bde4-1ebd-e1e9-2a3f-1f7a1a09])  //很重要


//全量数据
LOAD
 [a] as [b],
 [c] as [d]
FROM [lib://fft_qvd (win-i1nf3g039d9_qs_chanpin_dev)/用户数据201909_201912.qvd](qvd);

DROP TABLE [用户数据];
RENAME TABLE [用户数据_temp_da60bde4-1ebd-e1e9-2a3f-1f7a1a09] to [用户数据];


[autoCalendar]: 
  DECLARE FIELD DEFINITION Tagged ('$date')
FIELDS
  Dual(Year($1), YearStart($1)) AS [Year] Tagged ('$axis', '$year'),
  Dual('Q'&Num(Ceil(Num(Month($1))/3)),Num(Ceil(NUM(Month($1))/3),00)) AS [Quarter] Tagged ('$quarter', '$cyclic'),
  Dual(Year($1)&'-Q'&Num(Ceil(Num(Month($1))/3)),QuarterStart($1)) AS [YearQuarter] Tagged ('$yearquarter', '$qualified'),
  Dual('Q'&Num(Ceil(Num(Month($1))/3)),QuarterStart($1)) AS [_YearQuarter] Tagged ('$yearquarter', '$hidden', '$simplified'),
  Month($1) AS [Month] Tagged ('$month', '$cyclic'),
  Dual(Year($1)&'-'&Month($1), monthstart($1)) AS [YearMonth] Tagged ('$axis', '$yearmonth', '$qualified'),
  Dual(Month($1), monthstart($1)) AS [_YearMonth] Tagged ('$axis', '$yearmonth', '$simplified', '$hidden'),
  Dual('W'&Num(Week($1),00), Num(Week($1),00)) AS [Week] Tagged ('$weeknumber', '$cyclic'),
  Date(Floor($1)) AS [Date] Tagged ('$axis', '$date', '$qualified'),
  Date(Floor($1), 'D') AS [_Date] Tagged ('$axis', '$date', '$hidden', '$simplified'),
  If (DayNumberOfYear($1) <= DayNumberOfYear(Today()), 1, 0) AS [InYTD] ,
  Year(Today())-Year($1) AS [YearsAgo] ,
  If (DayNumberOfQuarter($1) <= DayNumberOfQuarter(Today()),1,0) AS [InQTD] ,
  4*Year(Today())+Ceil(Month(Today())/3)-4*Year($1)-Ceil(Month($1)/3) AS [QuartersAgo] ,
  Ceil(Month(Today())/3)-Ceil(Month($1)/3) AS [QuarterRelNo] ,
  If(Day($1)<=Day(Today()),1,0) AS [InMTD] ,
  12*Year(Today())+Month(Today())-12*Year($1)-Month($1) AS [MonthsAgo] ,
  Month(Today())-Month($1) AS [MonthRelNo] ,
  If(WeekDay($1)<=WeekDay(Today()),1,0) AS [InWTD] ,
  (WeekStart(Today())-WeekStart($1))/7 AS [WeeksAgo] ,
  Week(Today())-Week($1) AS [WeekRelNo] ;

DERIVE FIELDS FROM FIELDS [user_amount] USING [autoCalendar] ;

 

3.最后，跑个增量(20191201-20191210),代码如下

1.  select * from test where time >= '20190101' and where time >= '20191201'  
 

这个时候的加载，就会由两部分组成，全量(from)+增量，这里接不截图了。。。。