facebook architecture
Web front-end written in PHP. Facebook's HipHop Compiler then converts it to C++ and compiles it using g++, thus providing a high performance templating and Web logic execution layer. Because of the limitations of relying entirely on static compilation, Facebook's started to work on a HipHop Interpreter as well as a HipHop Virtual Machine which translate PHP code to HipHop ByteCode.
HipHop for PHP (HPHPc) is a PHP transpiler created by Facebook. By using HPHPc as a source-to-source compiler, PHP code is translated into C++, compiled into a binary and run as an executable, as opposed to the PHP's usual execution path of PHP code being transformed into opcodes and interpreted. The original motivation behind HipHop was to save resources on Facebook servers, given the large PHP codebase of facebook.com. Increases in web page generation throughput by factors of up to six have been observed over the Zend PHP.
HipHop Virtual Machine (HHVM), which is a just-in-time (JIT) compilation-based execution engine for PHP, also developed by Facebook. There were many reasons for this;
- one of them was HPHPc's flattened curve for further performance improvements.
- Also, HPHPc did not fully support the PHP language, including the create_function() and eval() constructs,
- it involved a specific time- and resource-consuming deployment process that required a bigger than 1 GB binary to be compiled and distributed to many servers in short order.
- In addition, maintaining HPHPc and HPHPi in parallel (as they needed to be, for the consistency of production and development environments) was becoming cumbersome.
- Finally, HPHPc was not a drop-in replacement for Zend, requiring external customers to change their whole development and deployment processes to use HPHPc
Business logic is exposed as services using Thrift. The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages.
Services implemented in Java don't use any usual enterprise application server but rather use Facebook's custom application server. At first this can look as wheel reinvented but as these services are exposed and consumed only (or mostly) using Thrift, the overhead of Tomcat, or even Jetty, was probably too high with no significant added value for their need.
Persistence is done using MySQL, Memcached, Hadoop's HBase. Memcached is used as a cache for MySQL as well as a general purpose cache.
Offline processing is done using Hadoop and Hive.The Apache Hive data warehouse software facilitates querying and managing large datasets residing in distributed storage. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL.
Data such as logging, clicks and feeds transit using Scribe and are aggregating and stored in HDFS using Scribe-HDFS, thus allowing extended analysis using MapReduce. Scribe is a server for aggregating log data streamed in real time from a large number of servers.
BigPipe is their custom technology to accelerate page rendering using a pipelining logic.
Varnish Cache is used for HTTP proxying. They've prefered it for its high performance and efficiency.
The storage of the billions of photos posted by the users is handled by Haystack, an ad-hoc storage solution developed by Facebook which brings low level optimizations and append-only writes.
Facebook Messages is using its own architecture which is notably based on infrastructure sharding and dynamic cluster management. Business logic and persistence is encapsulated in so-called 'Cell'. Each Cell handles a part of users ; new Cells can be added as popularity grows. Persistence is achieved using HBase.
Facebook Messages' search engine is built with an inverted index stored in HBase. Link
The typeahead search uses a custom storage and retrieval logic. Link
Chat is based on an Epoll server developed in Erlang and accessed using Thrift. Erlang is a programming language used to build massively scalable soft real-time systems with requirements on high availability. Some of its uses are in telecoms, banking, e-commerce, computer telephony and instant messaging. Erlang's runtime system has built-in support for concurrency, distribution and fault tolerance.
They've built an automated system that respond to monitoring alert by launching the appropriated repairing workflow, or escalating to humans if the outage couldn't be overcome.
About the resources provisioned for each of these components, some information and numbers are known:
- Facebook is estimated to own more than 60,000 servers. Their recent datacenter in Prineville, Oregon is based on entirely self-designed hardware that was recently unveiled as Open Compute Project.
- 300 TB of data is stored in Memcached processes
- Their Hadoop and Hive cluster is made of 3000 servers with 8 cores, 32 GB RAM, 12 TB disks that is a total of 24k cores, 96 TB RAM and 36 PB disks
- 100 billion hits per day, 50 billion photos, 3 trillion objects cached, 130 TB of logs per day as of july 2010