Facebook Scribe介绍

 

Facebook Scribe介绍

-------------------

1. 介绍

            Scribe是Facebook一个开源的实时分布式日志收集系统。它提高了大规模日志收集的可靠性和可扩展性。你可以在不同的节点上安装Scribe服务，然后这些服务会把收集到的信息发布到中心的服务集群上去。当中心服务不可得到时，本地的Scribe服务会暂时把收集到的信息存储到本地，等中心服务恢复以后再进行信息的上传。中心服务集群可以把收集到的信息写入本地磁盘或者分布式文件系统上，如hdfs，或者分发到另一层的Scribe服务集群上去。

 

Scribe提供了一种无阻塞的thrift服务，底层依赖libevent库和thrift库的支持。

 

2. 安装

        安装的方法在https://github.com/facebook/scribe可以找到，这里提几点注意的，第一是要注意一些依赖库的安装，特别是thrift的fb303，它提供了对thrift服务的监控与动态配置等操作。在thrfit源代码的contrib中可以找到，安装方法与安装thrift一样。第二个就是如果你安装好了Scribe，在运行其中的例子的时候会出来python的模块找不到，注意要配置一下PYTHONPATH这个环境变量。

3. 例子

在Scribe的源代码包中有examples这个目录，其中有三个例子。

第一个是简单的如何启动Scribe服务与发送消息给Scribe服务。

#Create a directory to log messages: mkdir /tmp/scribetest #Start scribe using the configuration in example1.conf: src/scribed examples/example1.conf #From another terminal, use scribe_cat to send a message to scribe: echo "hello world" | ./scribe_cat test #If the previous command failed, make sure you did a 'make install' from the #root scribe directory and that $PYTHONPATH is set correctly(see README.BUILD) #Verify that the message got logged: cat /tmp/scribetest/test/test_current #Check the status of scribe (requires root): ./scribe_ctrl status #Check scribe's counters (you should see 1 message 'received good'): ./scribe_ctrl counters #Shutdown scribe: ./scribe_ctrl stop 

第二个例子是如何在多个Scribe实例上进行消息的传递。

# This example shows you how to log messages between multiple Scribe instances. # In this example, we will run each Scribe server on a different port to simulate # running Scribe on multiple machines. 'client' 'central' ---------------------------- -------------------- | Port 1464 | | Port 1463 | | ---------------- | | ---------------- | | -> | scribe server |--|--->| | scribe server | | | ---------------- | | ---------------- | | | | | | | | | temp file | | | temp file | |--------------------------- |------------------- | ------------------- | /tmp/scribetest/ | ------------------- #Create a directory for the second scribe instance: mkdir /tmp/scribetest2 #Start up the 'central' instance of Scribe on port 1463 to write messages to disk #(See example2central.conf): src/scribed examples/example2central.conf #Start up the 'client' instance of Scribe on port 1464 to forward messages to #the 'central' Scribe server (See example2client.conf): src/scribed examples/example2client.conf #Use scribe_cat to send some messages to the 'client' Scribe instance: echo "test message" | ./scribe_cat -h localhost:1464 test2 echo "this message will be ignored" | ./scribe_cat -h localhost:1464 ignore_me echo "123:this message will be bucketed" | ./scribe_cat -h localhost:1464 bucket_me #The first message will be logged similar to example 1. #The second message will not get logged. #The third message will be bucketized into 1 of 5 buckets #(See example2central.conf) #Verify that the first message got logged: cat /tmp/scribetest/test2/test2_current #Verify that the third message got logged into a subdirectory: cat /tmp/scribetest/bucket*/bucket_me_current #Check the status and counters of both instances: ./scribe_ctrl status 1463 ./scribe_ctrl status 1464 ./scribe_ctrl counters 1463 ./scribe_ctrl counters 1464 #Shutdown both servers: ./scribe_ctrl stop 1463 ./scribe_ctrl stop 1464 ./scribe_ctrl stop  

第三个例子是当中心服务关闭以后，看本地的Scribe的一个缓冲反应。

# Test Scribe buffering #Startup the two Scribe instances used in Example 2. #Start the 'central' server first: src/scribed examples/example2central.conf #Then start the 'client': src/scribed examples/example2client.conf #Log a message to the 'client' Scribe instance: echo "test message 1" | ./scribe_cat -h localhost:1464 test3 #Verify that the message got logged: cat /tmp/scribetest/test3/test3_current #Stop the 'central' Scribe instance: ./scribe_ctrl stop 1463 #Attempting to check the status of this server will return failure since it not running: ./scribe_ctrl status 1463 #Try to Log another message: echo "test message 2" | ./scribe_cat -h localhost:1464 test3 #This message will be buffered by the 'client' since it cannot be forwarded to #the 'central' server. Scribe will keep retrying until it is able to send. #After a couple seconds, the status of the 'client' will be set to a warning message: ./scribe_ctrl status 1464 #Try to Log yet another message(which will also get buffered): echo "test message 3" | ./scribe_cat -h localhost:1464 test3 #Restart the 'central' instance: src/scribed examples/example2central.conf #Wait for both Scribe instance's statuses to change to ALIVE: ./scribe_ctrl status 1463 ./scribe_ctrl status 1464 #Verify that all 3 messages have now been received by the 'central' server: cat /tmp/scribetest/test3/test3_current #Shutdown: ./scribe_ctrl stop 1463 ./scribe_ctrl stop 1464 

 

由于没有找到c++的例子，这里自己写了一个简单的例子，仅供参考

#include <iostream> // scribe headers #include "scribe.h" // this header is in the gen-cpp directory // thrift headers #include <protocol/TBinaryProtocol.h> #include <transport/TSocket.h> #include <transport/TTransportUtils.h> using namespace std; using namespace boost; using namespace apache::thrift; using namespace apache::thrift::protocol; using namespace apache::thrift::transport; using namespace scribe::thrift; int main(int argc, char** argv) { std::string category; // the log category std::string host; // the ip host int port; // the port if(argc == 2) { category.assign(argv[1]); host = "localhost"; port = 1463; }else if (argc == 4 && strcmp(argv[1],"-h") == 0) { category.assign(argv[3]); char* sep = strstr(argv[2],":"); if(sep == NULL) { host.assign(argv[2]); port = 1463; } else { char tmp[20]; strncpy(tmp,argv[2],sep - argv[2]); port = atoi(sep+1); host.assign(tmp); } }else { printf("usage (message is stdin): scribe_cat [-h host[:port]] category"); exit(1); } shared_ptr<TTransport> socket(new TSocket(host, port)); shared_ptr<TTransport> transport(new TFramedTransport(socket)); shared_ptr<TProtocol> protocol(new TBinaryProtocol(transport)); // use boost shared pointer shared_ptr<scribeClient> sbclient(new scribeClient(protocol)); try { transport->open(); // create a log entry and push it to vector LogEntry le; le.message = "This is a test message"; le.category = category; std::vector<LogEntry> messages; messages.push_back(le); // write to scribe server sbclient->Log(messages); // close the transport transport->close(); }catch(TException &tx) { printf("ERROR: %s/n",tx.what()); } } 

 

 相关的Makefile文件内容：

 CC=g++ LIBS=-lthrift -lfb303 INCLUDE=-I/usr/local/include/thrift/fb303 -I/usr/local/include/thrift SOURCE=scribe_cat.cpp scribe.cpp scribe_types.cpp scribe_cat: $(SOURCE) $(CC) $(SOURCE) -o $@ $(INCLUDE) $(LIBS) clean: rm scribe_cat 

 

4. 参考

 

https://github.com/facebook/scribe

http://blog.nosqlfan.com/html/1287.html

http://incubator.apache.org/thrift/