Logging Enhancements in Microsoft Forefront Threat Management Gateway (TMG) 2010
Introduction
Access logging is a vital component for an edge security device. Accurately and reliably recording requests allowed and denied by the firewall is essential for auditing, troubleshooting, and usage reporting. In many organizations logging is necessary for regulatory compliance, and in the event of a security breach it is critical to performing forensic analysis. In this article I will share with you the improvements made to the logging infrastructure in Forefront Threat Management Gateway (TMG) 2010.
Local Database Logging
By default, TMG is configured to log to a local SQL 2008 Express database that is installed along with TMG. This by itself is a substantial improvement over previous versions of ISA which utilized MSDE. MSDE, which is officially known as the Microsoft SQL Desktop Engine (the operative word here being desktop), was never designed for serious enterprise use. If you have ever managed an ISA firewall infrastructure in a busy environment you are well aware of the limitations imposed by MSDE. Under load the MSDE database would quickly become a bottleneck. When the firewall was unable to write to the log database, the firewall service would shut down and all traffic would be denied. MSDE also had additional limitations, including a workload governor and a 2GB database file size limit. By comparison, SQL 2008 Express has a 4GB database file size limit, does not include a workload governor, and is much more robust and higher performing. It does have some limitations of its own, however. SQL Server 2008 Express is limited to a single CPU socket and 1GB RAM.
Remote Database Logging
TMG can be configured to log to a remote SQL server, which provides an alternative to the limitations imposed by the local SQL Server 2008 Express installation. There are advantages and disadvantages to this option. One significant advantage is having the ability to leverage core editions of SQL (Standard or Enterprise). Neither version of SQL has limitations on the amount of memory that can be used, nor does SQL Enterprise have a CPU limit (SQL Standard is limited to 4 CPUs). Of course this requires that you have a separate system and a license for SQL, but for most organizations this will not be a limiting factor. There are some drawbacks to using a remote SQL server, however. The native reporting tools in TMG will no longer work, as the TMG reporting tools rely on the locally installed SQL Reporting Services for operation. There are some excellent third-party reporting tools available as a replacement, or you could develop your own custom reports. Keep in mind that network connectivity between the TMG firewall and the remote SQL server has the potential to become a bottleneck. Ensure that you have abundant, reliable network bandwidth between the TMG firewall and remote SQL server for optimum performance.
Log Queuing
To further enhance the stability of the logging subsystem, TMG now includes a feature known as log queuing. This wonderful new capability makes logging much more resilient in times of heavy use. With log queuing, if the firewall service is unable to write to the log database or log file for any reason (e.g. connectivity to remote SQL server is disrupted, excessive disk activity, etc.), log data is buffered in the log queue in memory and written to a binary log file on disk. When connectivity to the database is restored or disk utilization allows writing to the text file, data from the log queue is written to the log.
Log queuing is enabled by default (there is no way to disable it) and other than specifying the location where the binary log files are stored, there isn’t anything else to configure. By default, the binary log files are placed in the Logs folder in the TMG installation folder. It is recommended that these log files be placed on a separate disk from the system partition to improve performance. The partition should have sufficient free disk space to handle extended periods of log database downtime. If you are running host-based anti-virus, be sure to exclude this folder from any real-time or scheduled scans.
To change the location of the Log Queue Storage folder, open the TMG management console and highlight the Logs & Reports node in the navigation tree.
Figure 1
In the Tasks pane on the right side, click the Configure Log Queue link.
Figure 2
To change the location of the log queue folder, select This folder (enter full path): and enter the new location.
Figure 3
Note:
When making this change on an Enterprise array, make certain this folder exists on all array members. For convenience you can also make use of a system variable here, for example %LOG_QUEUE_DIR%.
Additional Enhancements
In addition to the infrastructure improvements made to the logging facility, new log fields have been added to support the enhanced protection capabilities included in TMG. There is now additional information available about TMG Firewall Clients, Malware Protection, Network Inspection System (NIS), and URL filtering.
To view and select these new log fields, open the TMG management console and highlight the Logs & Reports node in the navigation tree. In the Tasks pane on the right side, click the Configure Web Proxy Logging link.
Figure 4
Select the Fields tab and scroll through the list of log fields. Highlighted are some of the new fields described previously.
Figure 5
Figure 6
Summary
Logging is one of the least exciting, yet most important features of an enterprise class firewall. Often overlooked until it fails or there is a security incident, the Forefront TMG team has recognized the importance of logging to the overall security solution and has invested time and effort in improving this critical infrastructure. Replacing MSDE with the more robust SQL 2008 Express makes the default logging option more reliable and efficient. The addition of log queuing to provide resilience makes remote SQL logging a viable alternative to local database logging.