http://www.tigase.net/blog-entry/1mln-or-more-onli
By admin on May 29, 2011
I have been working on clustering code improvements in the Tigase server for last a few months to make it more reliable and better scale. In article about XMPP Service sharding - Tigase on Intel ATOMs I have presented some preliminary results on a small scale.
In last weeks I had a great opportunity to run several tests over the Tigase cluster of 10 nodes on much better hardware. The goal was to achieve 1mln online users connected to the cluster generating sensible traffic. More tests have been run to see how the cluster behaves with a different number of connections and under a different load.
Below are charts taken from two tests. One test with top 1mln 128k online users and moderate traffic and the second with peak 1mln 685k online users and very reduced traffic.
All tests were carried out until the number of connections reached its maximum and for some time after that to make sure we receive a stable service when connections start dropping.
The test for 1mln online users run with a moderate traffic, that is a message from each online user every 400 seconds and status change every 2800 seconds.
The other test for 1mln 500k online users ran with no additional traffic except user login, roster retrieval, initial presence broadcast and offline presence broadcast on connection close.
The roster size for all online users was 150 elements of which 20% (30) were online during the test and new connections rate was 100/sec.
If you are interested in more details, please continue reading...
I guess the first question which comes to your mind is why so low traffic. Especially looking in presented charts there is for sure room for more.
The CPU would most likely handle more, probably at least as much as twice more traffic and memory usage shouldn't grow much either as traffic generates only temporary objects.
Indeed, the average traffic was estimated to a message every 200 seconds and presence broadcast every 20 mins on each user connection.
The "high" traffic was estimated to a message every 100 seconds and presence broadcast every 10minutes.
Unfortunately as always with load tests, the problem was with generating enough traffic. I used Tsung 1.3 for testing which did really good job simulating user connections from 10 other machines, however it just couldn't do more than that.
Test environment used for tests
I had 21 identical machines at my disposal for duration of the tests: 2 x Xeon Quad 2.0GHz, 16GB RAM, 750GB SATA HDD, 1Gb ethernet.
One machine running Ubuntu Server 9.04 used as a database with MySQL 5.1 installed and tuned for the test.
10 machines running Ubuntu Server 9.04 with Tigase server installed in cluster mode, with Linux kernel and GC settings tuned for the test. Tigase server in version from SVN with some not yet committed changes.
10 machines running Proxmox 1.3 and Debian 5 in virtual machines. Tsung 1.3 on Erlang R13B01 was used as traffic and load generator.
Results
As we can see on attached charts both tests were quite successful.
Of course nobody wants to run a service for 1mln 600k online users with idle connections. The second test was executed only to check the installation limits. As we can see on the memory chart the server completely used up memory. So with 16GB of RAM not much more is possible. Traffic was on quite stable level as it was only generated by new user connections in the first phase, then by both new connections and closing connections in the second phase, hence the CPU load jump, and by closing connections only in third phase.
Much more interesting charts are for the 1mln online users testwith traffic on each connection. We can clearly see "steps" on the cluster traffic chart and less clear steps on the session manager traffic chart. They are related to presences updates "wave" which was starting every 2800 seconds. The CPU usage stayed at about 60% at peak time with plenty of room for more traffic. Memory consumption was quite high at about 70% at peak number of connections.
Other tests
As I mentioned before I have run several tests to see how the server works under a different conditions. There is for sure no room here to present all charts, however I could post them if there is an interest for that. Please send me a message or add comments to the article if you want to see more charts.
The server was tested under different loads:
A message every 100 seconds and presence broadcast every 700 seconds on each connection.
A message every 200 seconds and presence broadcast every 1400 seconds on each connection.
A message every 400 seconds and presence broadcast every 2800 seconds on each connection.
A message every 800 seconds and presence broadcast every 5600 seconds on each connection.
No traffic except packets related to user login, roster retrieval, initial presence broadcast and offline presence broadcast.
Other tests I have run are listed below:
250k connections over plain TCP with load 1
250k connections over SSL with load 1
500k connections over plain TCP with load 1 and 2
500k connections over SSL with load 1, 2 and 3
750k connections over plain TCP with load 2 and 3
1mln connections over plain TCP with load 2 and 3
1mln 500k connections over plain TCP with load 5
Please note, given max number of connections is a target number, actual tests usually reached more.
Charts
All charts display plots for all 10 cluster nodes with a different colour for each node. In most cases only one plot (blue) is visible as user distribution was very even, hence load was the same. This is especially confusing for connections chart when all 10 plots look like a single blue line.
While chart plots display values for a particular node, the chart title displays sum for all nodes, the max is the maximum total registered by the monitor.
admin's blog
Log in to post comments