SecurityWarrior Consulting的Anton Chuvakin博士在去年底的时候写过一篇文章：Top 10 Things Your Log Management Vendor Won’t Tell You，很有意思。实际上，他提醒用户在选择日志审计产品，尤其是用它来做内控的目的的时候应该注意的问题。这些注意事项有助于帮助客户建立起合理的产品预期，也有助于督促日志审计/日志管理厂商去思考这些问题背后的解决方案。好吧，让我们大家都擦亮眼睛。
原文摘录如下：

While many people have seen the “Top 10 things that your chef, real-estate agent, wedding planner or pilot won’t tell you,” the world has not yet seen Top 10 things your log management vendor won’t tell you. Finally, this gap has been closed.

  • “We talk analytics, but really, most of our customers only use us for collection.” While some products within SIEM and log management offer advanced analytics features, many customers are not truly ready for them. They need to  start dealing with the basics—logging, log collection, log review—before delving into advanced areas. Buying a product based on features you won’t use is a mistake.
     
  • “Our tool won’t make you PCI compliant. You’d have to do A LOT of things yourself – every day – to get and maintain compliance.” Sadly, many security solutions—and SIEM / log management are no exception—are sometimes sold as “compliance in a box.” You need to be aware that to stay PCI compliant you need to do more than purchase tools.
     
  • “No, you cannot buy an entire SOC in this small box.” Just as with compliance, you cannot buy an entire Security Operations Center in a box, big or small. However, some will try to sell you their SIEM as “SOC-in-a-box.” Running an effective SOC includes multiple processes and procedures which are just as necessary as a market-leading SIEM tool.
  • “We are cloud-ready, because … mmmmm… well, we are ready for it!” Many vendors will tell you that their tools are cloud-ready – without really thinking about what they mean. Effectively monitoring traditional and multi-tenant cloud environments distributed across regions and countries requires more than updated marketing materials! As a customer, you will need to carefully test the tool in your own hybrid environment before concluding that it is “cloud ready.”
  • “Our SIEM is really just a renamed log management tool. But that’s all you probably need.” The confusion around SIEM and log management functionality rages on – it also allows some tools to be sold as SIEM without having any critical SIEM functionality such as correlation and real-time dashboards.  Even though it might be all many customers need, it does not make such tool a SIEM tool.
  • “We can do everything with logs, but it might require some SMALL customizations. Our PS team is standing by!” More than a few SIEM vendors will promise support for every possible log including logs they have never seen. However, fully integrating a new log source for reporting, correlation and visualization will always takes work and cannot be taken for granted.
  • “If you make a mistake with capacity planning, we’d be happy to sell you more log management than you really need.” Many organizations are having trouble estimating how much log data will be coming into their SIEM or log management tools.  Both underestimating and overestimating are common.  It is recommended that you spend about a week measuring log volumes across the systems that will be reporting to a SIEM.
  • “We think our tool is scalable, but we don’t really have production customers of your size. Our engineers believe that it might work.” Scalability claims are cheap and frequently made by SIEM and log management vendors. However, the only real proof that the tool will scale to your requirements is testing the tool in your environment. Thus, you should insist on performance testing during the pilot if there are any doubts.
  • “We estimate our performance using really small log message sizes.” Yes, our tools can do a million messages an instant – but these are our special messages that we create in the lab. Nowadays, application logs and the proliferation of XML-based logging has pushed message sizes up to 1 kb or more from the traditional 200 byte logs from firewalls.  Thus, you need to be wary of performance estimates based on such artificially short logs.
  • “Our tool offers predictive security intelligence. No, we don’t know what it means either – and we can’t really predict it.” SIEM is one of the most over-hyped and over-marketed security technologies out there. The only way to make sure that a particular tool will satisfy your requirements is too carefully spell out those requirements and then test the tool yourself.


读完这篇文章，我也是颇有体会。
的确，对于我接触到的目前国内大部分客户而言，使用日志审计/日志管理产品的主要用途就是收集日志，进行查询、统计和报表。关联分析几乎很少使用。一方面，关联分析功能是一个吃力难讨好的技术，要么就要做到满足用户期望80分以上，否则再做也没啥用。用户对于关联分析的期望往往较高，即要求分析能力强，又要求对普通管理员易用易懂。从技术层面来说，还有一段路要走，要看商业智能（Business Intelligence）技术发展到什么程度了。另一方面，即便有较强的关联分析功能，大部分用户也并不关注。对于他们而言，当前工作的重心还是在收集、存储、查询、统计上，因为这些功能对用户是切实有用的，是基本的功能点。我觉得这是当前LM产品的重点所在。实际上，即便是这些看似基本的功能点也隐藏着巨大的技术挑战。因为面对海量异构事件的收集、存储和查询，LM厂商们将必须将性能提升到一个用户可以接受的水平。

与Anton Chuvakin的观点差不多，我也认为对于用户而言，在考虑SOC之前最好先考虑SIEM，或者干脆先考虑LM。至少不要在考虑的SOC的同时忽略SIEM和LM。

既然对于用户而言，对于当前的日志审计/LM产品重点是考察收集、存储和查询统计，那么又如何去甄别各个厂商对此的宣传和技术参数呢？例如，最重要的一个技术参数叫做EPS（Event per Second），亦即每秒事件数。实际上，各个厂商在给出这个值的时候，其条件和内涵可能完全不同。首先，你需要知道这个值是在什么条件下获得的，至少要知道是什么CPU、多少内存、多少硬件资源的条件下获得的；可能的话，还要知道测试的基准日志源是什么样的，这些日志是单设备日志，还是多源日志，平均日志长度是多少？除此之外，你还需要知道EPS的内涵所指为何？是单纯收集上来的EPS?还是指收集上来且归一化后的EPS？抑或是收集上来、归一化并持久化存储后的EPS。内涵不同，LM产品的工作机制不同，进行EPS的数值比较可能没有什么意义。而往往，几乎不会有厂商主动告诉你这些。如果你比较Care这些，最好的方式是建立自己的测试基准，进行横向实际测试比较。所以，对于重要的客户，我比较强调PoC。

用户必须清楚的认识到，LM是一类管理系统，其运用必须遵循管理类系统的生命周期。简单的说，无论厂商如何说LM，用户都是清楚认识到实施LM的工作内容，并且这些工作有很多是你必须参与其中，无法逃脱的。例如，我们在上LM的时候，应该了解到日志源种类和类型、规划日志容量、设计查询统计模板，同时，配套的运维也需要建立起来。别幻想一听完厂商的产品介绍就认为有了这个产品一切都OK了。