Real-Time Graphing With Graphite

If you’re building a web-based app, it’s always a good idea to build some instrumentation into your app. That way, you can see how your app is behaving, and how your users are interacting with your app over time.

I’m sure everyone who reads my blog is familiar with Google Analytics for tracking page hits. But what about what’s happening inside your app? Right now? Do you know?

Graphite is one way to graph the stats that you add to your app. Combine it with (say) statsd from Etsy, and adding any stats you want is easy. (Read this blog post from Etsy if you want to learn more about measuring your app, and how to add support for Statsd to your app).

Normally, you’ll probably be interested in looking at graphs that show your stats over a period of hours or days (for trend analysis), and both Graphite and Statsd are sensibly tuned for that. But what if you want to see what’s happening now, in real time? I couldn’t find any clear instructions on how to do that elsewhere, so here’s my take on how to do it. I’m assuming you’re already familiar with installing both Statsd and Graphite, and that you’ve already had both up and running successfully with their default configurations.

Making Statsd Forward Data In Real Time

By default, Etsy’s Statsd collects the data sent from your apps, and forwards it on to Graphite’s data collector (known as Carbon) every 10 seconds. For real time, we need the data forwarded on every second. To do that, edit your Statsd config, adding the ‘flushInterval’ value:

{
  graphitePort: 2003
, graphiteHost: "localhost"
, port: 8123
, flushInterval: 1000
}

A value of 1000 tells Statsd to forward data on every second.

Making Graphite Store Data At One Second Resolution

Graphite’s default / sample configuration tells it to store incoming data at 60-second resolution; that allows us to look at the total stats recorded minute by minute, but we can’t drill down to see what happens second by second. To do that, we have to tell Graphite to store the data on a second-by-second basis.

Edit /opt/graphite/conf/storage-schemas.conf, and add the following clause:

[real_time]
priority = 200
pattern = ^stats.*
retentions = 1:34560000

This tells Graphite that we want all the data received from Statsd to be kept on a second-by-second basis for 400 days … plenty long enough for any sort of comparison you might need to do.

I found that, to get Graphite to start using this new storage definition, I had to go and delete the existing data files by doing:

$ rm -rf /opt/graphite/storage/whisper/stats*

Getting Graphite To Draw Real-Time Graphs

Now all we need to do is to get Graphite showing you all the collected data in real-time. By default, Graphite will happily plot the data onto a graph, but will only generate an updated graph every 60 seconds. That’s perfect for an ops team looking for trends over hours, but it isn’t real-time.

If you’re using Memcache with Graphite, you’ll need to add this to your /opt/graphite/webapp/graphite/local_settings.py file, to tell Graphite to only cache data in Memcache for 1 second:

MEMCACHE_DURATION = 1

Is it worth caching the data at all at this resolution? Honestly, I don’t know. I guess that depends on how many people need to watch the data in real-time or not. Ideally, it would be better if Graphite dynamically set the Memcache timeout based on the data stored in the particular key, but for now, you need to either stop using Memcache, or set the cache duration to 1 second.

This now gives you graphs with 1-second resolution … now we just need to change the Graphite web-app’s auto-refresh feature to load a new graph every second. By default, it will only generate an updated graph every 60 seconds. To change this, we have to edit some of the app’s Javascript code.

• Open the file /opt/graphite/webapp/content/js/composer_widgets.js, and locate the function ‘toggleAutoRefresh’.
• Locate the ‘interval’ variable inside that function. Change its value from ’60′ to ’1′.
• Save the file, then refresh your web browser page.

Et voila. If you switch on auto refresh, you should now be able to see your app’s data being plotted second by second, giving you a real-time view of what your app is doing.