Hbase counter
本文是本人在做项目中有一个需求，就是日志分析各字段，我是定时去分析日志，需要记录每个域名+时间戳的hits和bits，但是有一个问题处理起来比较头疼，那就是有的日志会有延迟，比如12点去跑11点的日志，但是发现11点的日志中有10点的日志，所以每次都是先去取出来 ，若存在，则累加，再放入，这效率太低了，之后发现counter正好可以完成该使用场景，即：read+count+write，正好完成，就是讲key的value读出，若存在，则完成累加，再写入，若不存在，则按“0”处理，再加上你需要累加的值。
Introduction to Counters
In addition to the check-and-modify operations you saw earlier, HBase also has a mechanism to treat columns as counters. Otherwise, you would have to lock a row, read the value, increment it, write it back, and eventually unlock the row for other writers to be able to access it subsequently. This can cause a lot of contention, and in the event of a client process, crashing it could leave the row locked until the lease recovery kicks in—which could be disastrous in a heavily loaded system.
The client API provides specialized methods to do the read-and-modify operation atomically in a single client-side call. Earlier versions of HBase only had calls that would involve an RPC for every counter update, while newer versions started to add the same mechanisms used by the CRUD operations—as explained in CRUD Operations—which can bundle multiple counter updates in a single RPC.
Before we discuss each type separately, you need to have a few more details regarding how counters work on the column level. Here is an example using the shell that creates a table, increments a counter twice, and then queries the current value:
hbase(main):001:0>create 'counters', 'daily', 'weekly', 'monthly'0 row(s) in 1.1930 secondshbase(main):002:0>incr 'counters', '20110101', 'daily:hits', 1COUNTER VALUE = 1hbase(main):003:0>incr 'counters', '20110101', 'daily:hits', 1COUNTER VALUE = 2hbase(main):04:0>get_counter 'counters', '20110101', 'daily:hits'COUNTER VALUE = 2
Single Counters
The first type of increment call is for single counters only: you need to specify the exact column you want to use. The methods, provided by HTable, are as such:
long incrementColumnValue(byte[] row, byte[] family, byte[] qualifier, long amount) throws IOException long incrementColumnValue(byte[] row, byte[] family, byte[] qualifier, long amount, boolean writeToWAL) throws IOException
Given the coordinates of a column, and the increment account, these methods only differ by the optional writeToWAL parameter—which works the same way as the Put.setWriteToWAL() method.
Omitting writeToWAL uses the default value of true, meaning the write-ahead log is active.
Apart from that, you can use them easily, as shown in Example 4-17.
Example 4-17. Using the single counter increment methods
HTable table = new HTable(conf, "counters");
long cnt1 = table.incrementColumnValue(Bytes.toBytes("20110101"),
Bytes.toBytes("daily"), Bytes.toBytes("hits"), 1);
long cnt2 = table.incrementColumnValue(Bytes.toBytes("20110101"),
Bytes.toBytes("daily"), Bytes.toBytes("hits"), 1);
long current = table.incrementColumnValue(Bytes.toBytes("20110101"),
Bytes.toBytes("daily"), Bytes.toBytes("hits"), 0);
long cnt3 = table.incrementColumnValue(Bytes.toBytes("20110101"),
Bytes.toBytes("daily"), Bytes.toBytes("hits"), -1);
-
Increase the counter by one.
-
Increase the counter by one a second time.
-
Get the current value of the counter without increasing it.
-
Decrease the counter by one.
The output on the console is:
cnt1: 1, cnt2: 2, current: 2, cnt3: 1
Just as with the shell commands used earlier, the API calls have the same effect: they increment the counter when using a positive increment value, retrieve the current value when using zero for the increment, and eventually decrease the counter by using a negative increment value.
Single Counters
例子：
//加1
long cnt1=table.incrementColumnValue(Bytes.toBytes("row1"),
Bytes.toBytes("f1"), Bytes.toBytes("c1"),(long)1);
//当前值
long cnt2=table.incrementColumnValue(Bytes.toBytes("row1"),
Bytes.toBytes("f1"), Bytes.toBytes("c1"),(long)0);
//减1
long cnt3=table.incrementColumnValue(Bytes.toBytes("row1"),
Bytes.toBytes("f1"), Bytes.toBytes("c1"),(long)-1);
Multiple Counters
在初始化Increment时，需要提供一个row key :接下来调用table.increment(Increment)方法时，所有的counters都在该行的cell中操作 Increment increment1 = new Increment(Bytes.toBytes("20110101"));
increment1.addColumn(Bytes.toBytes("daily"), Bytes.toBytes("clicks"), 1);
increment1.addColumn(Bytes.toBytes("daily"), Bytes.toBytes("hits"), 1);
increment1.addColumn(Bytes.toBytes("weekly"), Bytes.toBytes("clicks"), 10);
increment1.addColumn(Bytes.toBytes("weekly"), Bytes.toBytes("hits"), 10);
Result result1 = table.increment(increment1);
for (KeyValue kv : result1.raw()) {
System.out.println("KV: " + kv +
" Value: " + Bytes.toLong(kv.getValue()));
}
Increment increment2 = new Increment(Bytes.toBytes("20110101"));
increment2.addColumn(Bytes.toBytes("daily"), Bytes.toBytes("clicks"), 5);
increment2.addColumn(Bytes.toBytes("daily"), Bytes.toBytes("hits"), 1);
increment2.addColumn(Bytes.toBytes("weekly"), Bytes.toBytes("clicks"), 0);
increment2.addColumn(Bytes.toBytes("weekly"), Bytes.toBytes("hits"), -5);
Result result2 = table.increment(increment2);
for (KeyValue kv : result2.raw()) {
System.out.println("KV: " + kv +
" Value: " + Bytes.toLong(kv.getValue()));
}
-
Increment the counters with various values.
-
Call the actual increment method with the earlier counter updates and receive the results.
-
Print the
KeyValueand returned the counter value. -
Use positive, negative, and zero increment values to achieve the desired counter changes.
When you run the example, the following is output on the console:
KV: 20110101/daily:clicks/1301948275827/Put/vlen=8 Value: 1 KV: 20110101/daily:hits/1301948275827/Put/vlen=8 Value: 1 KV: 20110101/weekly:clicks/1301948275827/Put/vlen=8 Value: 10 KV: 20110101/weekly:hits/1301948275827/Put/vlen=8 Value: 10 KV: 20110101/daily:clicks/1301948275829/Put/vlen=8 Value: 6 KV: 20110101/daily:hits/1301948275829/Put/vlen=8 Value: 2 KV: 20110101/weekly:clicks/1301948275829/Put/vlen=8 Value: 10 KV: 20110101/weekly:hits/1301948275829/Put/vlen=8 Value: 5
When you compare the two sets of increment results, you will notice that this works as expected.
// 创建表，存在则不进行任何操作
String domaintablename = "logfilter_domain_minf_data";
String siptablename = "logfilter_sip_minf_data";
String cf = "db";
String[] reducekey,reducevalue;
try {
HBaseService domainhbaseService = new HBaseService(domaintablename);
domainhbaseService.creatTable(domaintablename, new String[] { cf });
HBaseService siphbaseService = new HBaseService(siptablename);
siphbaseService.creatTable(siptablename, new String[] { cf });
reduceoutputkeyset = reduceoutput.keySet();
Increment increment;
for(String reduceoutputkey : reduceoutputkeyset){
reducekey = reduceoutputkey.split("\\|");
System.out.println("keysize:"+reducekey.length);
System.out.println("key:"+reducekey[0]);
if(reducekey[0].equals("domain")){
reducevalue = reduceoutput.get(reduceoutputkey).split("\\|");
increment = new Increment(Bytes.toBytes(reduceoutputkey.substring(7,reduceoutputkey.length())));
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes("h"), Long.parseLong(reducevalue[0]));
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes("b"), Long.parseLong(reducevalue[1]));
domainhbaseService.getHtable().increment(increment);
}else if(reducekey[0].equals("sip")){
reducevalue = reduceoutput.get(reduceoutputkey).split("\\|");
increment = new Increment(Bytes.toBytes(reduceoutputkey.substring(4,reduceoutputkey.length())));
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes("h"), Long.parseLong(reducevalue[0]));
increment.addColumn(Bytes.toBytes(cf), Bytes.toBytes("b"), Long.parseLong(reducevalue[1]));
siphbaseService.getHtable().increment(increment);
}
}
domainhbaseService.closeConnection();
siphbaseService.closeConnection();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}