[awk点滴]对某一列数据去重

# tail -100 invoice.log 
2014-08-05 10:00:03,105 - invoice - ERROR - 22038 - total_amount_is_0| failure |bill_confirm id is 1351
2014-08-05 10:00:03,159 - invoice - ERROR - 22038 - total_amount_is_0| failure |bill_confirm id is 1353
2014-08-05 10:00:03,219 - invoice - ERROR - 22038 - total_amount_is_0| failure |bill_confirm id is 1385
2014-08-05 10:00:03,252 - invoice - INFO - 22038 - This trans to tax total have 13 bill_confirms
2014-08-05 10:10:02,494 - invoice - ERROR - 22281 - total_amount_is_0| failure |bill_confirm id is 1221
2014-08-05 10:10:02,556 - invoice - ERROR - 22281 - total_amount_is_0| failure |bill_confirm id is 1222
....
2014-08-05 15:00:02,456 - invoice - INFO - 28977 - send bill_confirm: 1410 success
2014-08-05 15:00:02,685 - invoice - INFO - 28977 - send bill_confirm: 1411 success
2014-08-05 15:00:02,921 - invoice - INFO - 28977 - send bill_confirm: 1412 success
2014-08-05 15:00:03,135 - invoice - INFO - 28977 - send bill_confirm: 1413 success
2014-08-05 15:00:03,371 - invoice - INFO - 28977 - send bill_confirm: 1414 success
2014-08-05 15:00:03,602 - invoice - INFO - 28977 - send bill_confirm: 1415 success
2014-08-05 15:00:03,846 - invoice - INFO - 28977 - send bill_confirm: 1416 success
2014-08-05 15:00:03,908 - invoice - INFO - 28977 - This trans to tax total have 7 bill_confirms
2014-08-05 15:10:02,332 - invoice - INFO - 29240 - This trans to tax total have 0 bill_confirms

日志的格式
需要得到今天失败的单据的行的id
2014-08-05 10:10:02,556 - invoice - ERROR - 22281 - total_amount_is_0| failure |bill_confirm id is 1222 
也就是最后的1222，这是每隔10分钟发送一次，取出所有failure的行之后，取出所有的 id然后除重才能得到今天失败的行！


最后的命令是：  

# awk '$1=="2014-08-05" {print $11, $15}' invoice.log|grep failure|awk '!a[$NF]++'|wc -l
13

参考： http://www.361way.com/awk-del-repeat-line/1692.html

对某列去重  awk '!x[$0]++' filename 

本文出自 “orangleliu笔记本” 博客，请务必保留此出处  http://blog.csdn.net/orangleliu/article/details/38386257