Nine tips for a healthy "in production" ASP.NET application

Referred from: http://blogs.msdn.com/dougste/archive/2006/04/05/568671.aspx

 

The following are based on problems I see people having every day in production ASP.NET systems.

1. Before you go into production, long before, plan, specify, architect, design, code and test for your application to meet your requirements for performance, scalability, monitorability, diagnosability. If your requirements are anything like my requirements they equal a quiet life and long coffee breaks.  As a starting point the patterns and practices guide Improving .NET Application Performance and Scalability provides an excellent starting point.

2. Ensure that the debug="false" on the <compilation> element in the web.config file of each and every ASP.NET application on the server. The default during development is "true" and it is a common mistake to allow this development time setting to find its way onto production servers during deployment. You don't need it set to true in production and it often leads to memory overhead and inefficiencies.

3. Wherever possible strong name and install to the global assembly cache (GAC) any assemblies that are used by more than one ASP.NET application. This will reduce memory consumption. In ASP.NET 1.1, do not deploy strong named assemblies to the BIN directory (i.e. if they are strong named make sure you DO put them in the GAC). Note that if assemblies are going in the GAC they need to be designed to operate in the shared AppDomain and you have to trust whoever wrote them.

4. If your ASP.NET application calls web services be aware that the default configuration of HttpWebRequest allows only 2 concurrent outgoing HTTP connections to a particular endpoint at a time (to comply with the specification). For example if you have 25 concurrent ASP.NET request going on, all making web service calls to the same web service, only two will actually be doing anything. Misconfiguration of this causes major scalability headaches. Start by reading this article.

5. Be careful using things that generate dynamic assemblies behind the scenes. These can include RegEx, XslTransform and XmlSerializer. For example if you create XmlSerializers using contructors that are documented as "for internal use only" you will get into trouble because the dynamic assemblies that are created do not get cached and re-used. In a production system this can lead to major memory leaks and bring a server down. For RegEx and XslTransform, try to reuse them wherever you can.

6. In IIS6 make good use of application pools and configure their performance and health monitoring capabilities sensibly. You don't have to have all your applications configured to run in the DefaultAppPool. Putting applications into different application pools allows you to do things like isolate an application that has particular memory usage or stability issues, or allow one particular application to use a version of the .NET runtime that your other applications are not currently compatible with.  You should also pay attention to the configuration of application pools. By default application pools are configured to recycle after 1740 minutes. This is 29 hours. This means that your application pool might recycle in the middle of the night one day and then in the middle of your peak online shopping time the next day. As a better alternative, configure the application pool to recycle at a specific (quiet) time of day.

7. Keep an eye on your event logs. This is where important and critical messages get logged. There is a good list available for ASP.NET. Note that some of the most importand ones to watch out for (like 1000 and 1003) are only applicable if you are using IIS5 or using IIS6 in IIS5.0 compatability mode. If you are using IIS6 in native mode then the management of the process hosting ASP.NET code is taken over by IIS6 itself and the aspnet_wp.exe worker process is no longer used. In that case IIS6 will log events relating to application pool recycles and terminations. It is important to note however that IIS does not log all such events by default. This article explains how to configure what events are logged. The default value for the metabase value LogEventOnRecycle is 137 which means it only logs an event for recycles due to defined memory thresholds being reached (by default there are none) or due to an elapsed time having passed (by default 1740 minutes). I always recommend setting this to 255 for all application pools. 

The other thing that sometimes happens when you start looking at event logs is that you discover that your application developers have been over enthusiastic in spewing out events that or often not very interesting. It is important to log events when things go wrong by be aware that you can sometimes flood the event log thus masking important messages. I would recommend creating your own event log when you install the application and log your events to that. Also consider giving your application configurable "levels" of event log. So by default you would only log events when something really bad happens but you build in the facility to increase the verbosity and detail of the events when you are investigating a specific issue.  There are a number of predefined logging and tracing facilities around like the logging application block and log4net.

8. Monitor performance from time to time, especially in the early weeks after "go live". Things may have gone great during all your performance and scalability testing (you did do performance and scalability testing right?) but the real world is always different. A very useful article to get started is ASP.NET Performance Monitoring, and When to Alert Administrators. It is amazing how often it is discovered after a system has gone live that it is using huge amounts of memory or throwing a ridiculous number of exceptions.

9. Deploy updates to your applications at an appropriate time of day. It makes sense deploy updates to your site at a time when traffic is low, like 3am. Of course many web sites (especially internet facing ones) are live 24 hours per day but ever site has its quiet times. I also recommend stopping the application pool (or IIS in the case of IIS5), deploying all your updates and restarting the application pool/IIS. Although ASP.NET shadow copying allows the "hot" deployment of assembly and page updates these will generally trigger an application restart within the worker process or application pool. That's all very well if your application has a normal memory footprint of 20Mb and is the only application on the server but what if it uses 300Mb and the other applications use 700Mb. In "steady state" that is fine but if the 300Mb application goes through a restart this could temporarily push the overall memory usage in the process beyond a critical level.