基于FMS的RTMP负载均衡实现方法

http://www.adobe.com/devnet/adobe-media-server/articles/authorization-access-plugins.html

The main purpose of Adobe Flash Media Server (FMS) is to deliver multimedia content to users. FMS also features a handy tool for building the business logic of real-time, multiuser applications: the built-in scripting language, Server Side ActionScript. SSAS has almost everything you need to develop applications (except built-in database support).

However, if you want to go beyond SSAS to extend the functionality of Flash Media Server, you have a more powerful option: custom modules or plug-ins. Plug-ins can help you accomplish the following tasks:

• Monitor more closely what is happening inside the server
• Detect and handle events
• Monitor the load on various server components
• Monitor inbound connections
• Gain access to content located on any distributed file system

These are but a few of the things you can build Flash Media Server plug-ins to do. In this article, I'll give you the conceptual framework to understand how to build plug-ins and show you a few examples.

Understanding the Flash Media Server architecture

Flash Media Server is based on a client-server architecture. By default, any installation includes at least four processes (see Figure 1):

• fmsmaster
• fmsedge
• fmscore
• fmsadmin

Figure 1. The FMS architecture allows fmsmaster and fmsedge to hand off responsibilities to plug-ins for connection control, authorization, and file access.

When Flash Media Server starts, fmsmaster is launched. This process is mainly used to start or stop the core processes when necessary. You can run only one fmsmaster, and you cannot manage or configure it. Client connections are never handled by this process.

All inbound client connections are initially captured by the fmsedge process, regardless of the FMS configuring scheme you use (Origin or Edge/Origin). Below you may find a description of the client connection which could be controlled by the Access plug-in, if one exists. After that, the client connections are handed over to the core processes running SSAS applications, the Authorization plug-in(s), and the File plug-in.

The fmsadmin process provides access to the Administration API, which enables you to monitor, configure, and manage FMS.

Flash Media Server plug-ins

For plug-in creation, Adobe has provided code samples to develop the following types of plug-ins:

• Access plug-in: This is another level of FMS security, processing inbound connections before they reach SSAS applications. The FMS server can run only one access plug-in. The main purpose of this type of plug-in is to accept or reject incoming connections based on some criteria: statistics data, an authorization request to an external entity, or other condition. Also, the Access plug-in allows you to differentiate the level of access to files and folders with content. For resource-intensive services, perhaps the most critical feature is that the Access plug-in can link incoming connections to specific core processes. This balances the server load and optimizes resource utilization. For more detail on distribution of inbound connections exemplified by load balancing, please refer to the "Using the Access plug-in for load balancing" section.
• Authorization plug-in: This enables you to manage events that occur within the core processes. It can control inbound connections, manage content access, and map between the logical and physical path to the content. Moreover, you can build sophisticated business logic to manage subscription-based access to content and accumulate statistics on content use. Also, you can asynchronously call SSAS application methods, disconnect clients from the application, and perform other functions. You may have several such plug-ins, and they will all handle FMS events in alphabetical order based on the plug-in filenames.
• File plug-in: This enables you to organize asynchronous access to content located at any distributed file system using any protocol. Also, you can control content caching based on your own rules. FMS can use only one file plug-in.

For more details, please refer to the following article and resources:

• Large-scale streaming deployments with Flash Media Interactive Server 3.5
• Adobe Flash Media Server 4.5 Technical Overview
• Adobe Flash Media Server 4.5 Developer's Guide

Using Authorization plug-ins and URL tokenization to control content access

Authorization plug-ins are designed to authorize client application access to the internal Flash Media Interactive Server event model. Such plug-ins are run within the fmscore process after the connection to the client application is established but not yet accepted by the core process.

A server can host one or more Authorization plug-ins. For example, one plug-in can be used to authorize content playback and another one to check a permission to publish content. The fmscore process loads and runs the plug-ins in alphabetical order based on plug-in filenames; so if plug-ins A and B are subscribed to the E_PLAY event, plug-in A will be the first to process it. Plug-in A may lock this event, preventing plug-in B from processing it. Still, if plug-in A processes the event without locking it, plug-in B has a chance to affect processing by locking the event.

For more information on event types and fields available for each Authorization plug-in event, please refer to Developing an Authorization plug-in in the Flash Media Server Developer's Guide.

General recommendations

Events of the Authorization plug-in are classified into two types:

• Notify events are those that do not block the thread inside the core process. The core thread simply informs the plug-in(s) of a certain event and continues.
• Authorize events are those blocking the thread. In this case, a thread waits for the plug-in's immediate response to the event, and then continues depending on the response.

When processing both types of events, please make sure to call the specific service methods that will inform the core process that the event was accepted by the plug-in and is no longer needed. For notify events, use the onNotify()method. If you do not call this method for each accepted event, the event will remain idle in the internal core process structures and cause a memory leak. For authorize events, use the onAuthorize() method—otherwise, the situation will be even worse than with the notify events. As authorize events are all blocking, then sooner or later FMS will no longer accept incoming connections, while current, already-connected clients will hang because all the running core threads will have to wait for event authorization.

It is also worth noting that the Authorization plug-in applies globally to all SSAS applications run by FMS. If you want the plug-in to accept events only for certain applications, you can do it as follows. The plug-in should be aware of the SSAS application name (for instance, from a configuration file). Then you have to pass to the event handler the name of the SSAS application to which the event was fired and compare that name with the target application. Here is an example:

void MyFmsAuthorizeEvent:: authorize () {      / / Target SSAS application for which the plug-in is run      std:: string our_app_name = "vod";      bool allowed = true;      switch (m_pAev-> getType ()) {          case IFmsAuthEvent:: E_PLAY: {              / / Get the name of the SSAS application to which the event was fired              / / It may also contain names of instances, etc.              std:: string app_name = getStringField (m_pEvent, IFmsAuthEvent:: F_APP_NAME);              / / Get the "clean" name of the SSAS application              size_t dash_pos = app_name.find ('/');if (dash_p os!= Std:: string:: npos)                    app_name = app_name.substr (0, dash_pos);              / Is this event targeted to our application?              if (app_name!= Our_app_name)                    break;              / / Otherwise handle the event              . . .              allowed = ...;          }      }      pServerCtx-> onAuthorize (m_pEvent, allowed, NULL);}

If an event takes a long time to process—for example, it requires making some calculations or sending a request to an external system—it is recommended to allocate a special thread pool and delegate event handling to threads in this pool.

RTMP URL tokenization

To improve the safety of access to content, tokenization and obfuscation of the path to content located at the server is used. This may be required for various reasons: by copyright holders, for instance, to prevent intruders from deriving the full path to content on the server, or to limit access to content by using tokens. The token can be a random string that uniquely identifies the content unit or an encrypted path to content or access permissions of a certain group of users, or any other suitable criteria.

Here I'll describe the implementation of this functionality for two popular FMS deployment models. The first configuration consists of Origin servers only; the second configuration includes both Edge and Origin servers.

In this example, the plug-in just converts the content's logical path to its physical path, usually derived from the content name (a token in this case) provided by the client application.

To change the physical path to the content, you need to change the F_STREAM_PATH field, which is read-only for several events but is read-write for the E_FILENAME_TRANSFORM event only. This event is generated immediately after the E_PLAY event when the core process attempts to map the content's logical path to its physical path. This event will be fired until the content is found at the logical path, or all possible content sources are looped through.

From now on I'll focus on the authorize events, which block the thread in the core process to completely control the FMS operation.

Origin-only scheme

In this configuration, the Authorization plug-in for tokenization is quite simple: it subscribes to E_FILENAME_TRANSFORM event and changes the logical path to the physical path (see Figure 2).

Figure 2. Configuration based on Origin servers only

First, the plug-in must subscribe to the E_FILENAME_TRANSFORM event only:

void FmsAuthAdaptor:: getEvents (I32 aevBitAuth [], I32 aevBitNotf [], unsigned int count) {      . . .      / / Unsubscribe from all events except the one you need           IFmsAuthEvent:: EventType authExcludeEvent [] = {          IFmsAuthEvent:: E_APPSTART,          IFmsAuthEvent:: E_APPSTOP,          IFmsAuthEvent:: E_CONNECT,          IFmsAuthEvent:: E_DISCONNECT,          IFmsAuthEvent:: E_PLAY,           / / IFmsAuthEvent:: E_FILENAME_TRANSFORM,           IFmsAuthEvent:: E_STOP,          IFmsAuthEvent:: E_SEEK,          IFmsAuthEvent:: E_PAUSE,          IFmsAuthEvent:: E_PUBLISH,          IFmsAuthEvent:: E_UNPUBLISH,          IFmsAuthEvent:: E_LOADSEGMENT,          IFmsAuthEvent:: E_ACTION,          IFmsAuthEvent:: E_CODEC_CHANGE,          IFmsAuthEvent:: E_RECORD,          IFmsAuthEvent:: E_RECORD_STOP      };      m_pFmsAuthServerContext-> excludeEvents (aevBitAuth, count, authExcludeEvent, sizeof (authExcludeEvent) / sizeof (authExcludeEvent [0]));      . . .}

Next, the plug-in gets the content token name from the event handler, derives the physical path, and changes the logical path to the physical path:

void MyFmsAuthorizeEvent:: authorize () {      bool allowed = false;      switch (m_pAev-> getType ()) {          case IFmsAuthEvent:: E_FILENAME_TRANSFORM: {              / / Get the content name              std:: string stream_name = getStringField (pEvent, IFmsAuthEvent:: F_STREAM_NAME);              / / Derive the physical path based on the token received              const char * real_path = . . .              / / Have you found a token and derived the physical path?              if (!real_path) {                    / / Write something to the log . . .                    break;              }              / / Change the physical path              if (!setStringField (pEvent, IFmsAuthEvent:: F_STREAM_PATH, real_path)) {                    / / Write something to the log                    break;              }              allowed = true;               }      }      pServerCtx-> onAuthorize (pEvent, allowed, NULL);}

There is a small nuisance with the E_FILENAME_TRANSFORM event. If you block the first event in the chain of events fired for every possible content source, all the other events will still fire, although there is no physical content at the token received.

Basically, this is not critical if the derivation of the physical path is not resource-intensive, involving no request to an external system. Otherwise, this problem can be circumvented by additionally subscribing to the E_PLAY event in order to make a primary validation of the token and authorize or block the event. Yet another way, after the first failure of the physical path derival from the token, is to forcibly substitute the physical path with the special (alternative) content.

Therefore, in current FMS versions, blocking the E_FILENAME_TRANSFORM event does not make much sense.

Edge/Origin scheme

It is well known that, in this configuration, the Edge server usually runs as a caching proxy. In this case, the Edge server first attempts to find content in the file cache and, if the content is not found (for instance, the needed fragment is missing), the Edge server requests it from the Origin server. For this example, assume that the file cache is enabled on the Edge server.

Basically, for this configuration the functionality described in the previous step is quite sufficient, and the plug-in can be installed on the Origin server only. Why only there? Because in this configuration no E_FILENAME_TRANSFORM events will be fired for the Edge server core processes. This is quite logical, since the SSAS applications (and, accordingly, the configuration files setting virtual directories) are run on the Origin server.

When it receives a request to play back the content, the Edge server will fail to find a proper content fragment in the file cache because the content is no more than a token. Then the Edge server will send a request to the Origin server, which will convert the physical path to the content based on the token (content name). The reply, along with a content fragment, will be sent back to the Edge server, which will be informed of the changes in the physical path to the content. This information is sent from the Origin server, as it is used to enable the file cache on the Edge server. Then the Edge server will either continue to request content fragments from the Origin server or use the previously received content from the file cache. This communication is shown in Figure 3.

Figure 3. Messaging between the Edge and Origin servers

So that the task might not seem trivial. In addition to content name tokenization, this example implements geographic IP-based filtering of incoming content requests. This is required by almost all copyright holders to meet video content licensing conditions.

It would be better to make such geographic filtering at the Access plug-in level, but as you have the already-operable Authorization plug-in, you can add it here. Also, this will ensure that any requests for content playback never avoid geo filtering!

This produces the model shown in Figure 4.

Figure 4. Plug-ins to enable geographic filtering on Edge and tokenization on Origin

The Edge server plug-in will perform the following:

• Initial validation of tokens
• IP-based geographic filtering of requests for each unit of content

The Origin server plug-in will perform the following:

• Conversion of the physical path to the content based on the token received (the name of the content received from the client application)

The Edge server plug-in must do the following:

• Subscribe to the E_PLAY event. The related code has already been given above.
• Validate the token and apply geographic filtering of requests based on IP address, using the event handler:

void MyFmsAuthorizeEvent:: authorize () {      bool allowed = false;      switch (m_pAev-> getType ()) {          case IFmsAuthEvent:: E_PLAY: {              / / Validate the token              / / Get the content name              std:: string stream_name = getStringField (pEvent, IFmsAuthEvent:: F_STREAM_NAME);              if (!validate_token (stream_name)) {                    / / Write something to the log ...                    break;              }              / / Get the IP address              std:: string cip = getStringField (pEvent, IFmsAuthEvent:: F_CLEINT_IP);              / / Verify IP address permission to access the content              if (!content_available_for_ip (stream_name, cip)) {                    / / Write something to the log                    break;              }              / / Authorize the event              allowed = true;          }      }      pServerCtx-> onAuthorize (pEvent, allowed, NULL);}

It is clear that token validation and access to content can be combined into one operation, depending on your system architecture.

The plug-in for Origin server fully repeats the code specified in the "Origin-only scheme" section.

While developing Authorization plug-ins, give particular attention to optimizing your source code, which is responsible for event processing performance. Also, please do not forget to notify the core process that an event requires authorization or is no longer needed. Also, heavy transactions with external systems should be delegated to special threads with pooling and timeouts.

Using the Access plug-in for load balancing

The Access plug-in is another level of Flash Media Server security that allows you to perform lightweight authorization of incoming connections before they reach the level of SSAS applications in the fmsedge process.

The Access plug-in provides the following features:

• Analyzes the properties of a client connection
• Analyzes server statistics
• Differentiates access to files and folders with content
• Sends requests to external systems

Once a client connection reaches the fmsedge process, fmsedge passes connection control to the Access plug-in (if installed). The Access plug-in business logic can now decide on how to handle this connection: accept, reject, or forward it to another FMS server.

Distributing FMS load among its fmscore processes

One of the most fascinating features offered by the Access plug-in is the ability to make a dynamic selection of fmscore process to which to forward a connection, right at the moment when the inbound connection arrives to fmsedge. This helps to manage client connections at the FMS level more or less flexibly, according to your system architecture and your resources. For example, this way you can guarantee the quality of service to certain groups of customers.

The process of binding the connection to an fmscore process is well defined in Assigning an application to a core processin the Flash Media Server Developer's Guide. Here's an example of obtaining the fmscore process ID to which to attach a connection obtained from the client connection URI. It is clear that originally this identifier should be added to the URI, for instance within the player used to play back the content. The player could retrieve the ID from the load balancer or calculate it based on the parameters of the requested content. This example will be used later in this article when discussing load balancing with SSAS applications:

void MyAdaptor::onAccess(IFCAccess* pAccess) {    switch(pAccess->getType()) {        case IFCAccess::CONNECT: {            // get client's URI            const char* uri = pAccess->getValue("s-uri");            const char* tmp = 0;            // marker where core id is in the URI            const char *pName = "coreId=";            if(uri && (tmp=strstr(uri, pName))!=0) {                // get a fmscore id                const char *coreId = tmp + strlen(pName);                // set the fmscore id                if(!pAccess->setValue("coreIdNum", coreId)) {                    // log that set is failed                }            }            break;        }    }}

Video-on-demand content load balancing

A good example of using the Access plug-in is a load balancing system implemented for a video-on-demand content broadcast platform. Please note that this is just one of the possible options; still, it is quite functional and is used in video portals with large audiences. However, the efficiency of the architecture certainly depends on the content type.

Suppose you have a pool of Edge servers with the FMS server providing access to the video-on-demand content using a standard SSAS VoD application. The Edge servers receive content from the Origin servers that have a direct access to the video-on-demand content storage. This means that Edges should have proxying enabled so they can keep part of the content in the local cache on the disk.

The goal is to develop a more or less intelligent load balancer for a video platform that will reduce the server load. Reducing the load assumes, for each Edge server, efficient utilization of its resources, such as cache, network interface bandwidth, CPU, and so on.

For this purpose, I have developed two SSAS applications and an Access plug-in. The solution architecture is shown in Figure 5. On the Edge server, you would install the Access plug-in to attach the inbound client requests to the proper fmscore process. At the same location, you'd install the edge_app SSAS application, which will collect statistics related to the file cache, the number of active users, and so on, by calling the FMS AdminAPI methods. Another SSAS application, balancer_app, will be responsible for initial distribution of user requests to play back certain pieces of content.

Figure 5. Video-on-demand content balancing architecture

So the client must initially access the load balancer, passing the content name to it. In response, the load balancer will get the Edge Server ID and fmscore process ID to which the Access plug-in will attach the client request. The load balancer periodically collects statistics (file cache content , performance characteristics, and so on) from the Edge servers. Based on these statistics, it distributes the arriving client requests using the RTMP redirect method.

Edge_app SSAS application

This SSAS application periodically collects statistics from FMS and its cache. The statistics are gathered by calling the FMS AdminAPI methods. For purposes of this example, you need only two methods: getFileCacheStats() and getServerStats(). The first method provides statistics on the file cache in RAM grouped by the fmscore processes. The second method provides the server statistics: the number of active connections, bandwidth allocation, CPU utilization, and so on.

Also, the application configuration file can accommodate the physical server characteristics which the load balancer can factor in to distribute the queries: the number of fmscore processes, the maximum server throughput, and so on.

All these characteristics and properties are processed and cached to be used by the balancer_app SSAS application. For example, you can dump statistics from the Edge server like this:

'srvStats' => {     'appURI' => 'rtmp://192.168.1.10/vod',     'maxOutBandwidth' => 6250000,     'cpuUsage' => 0,     'memoryUsage' => 24,     'phisicalMem' => 35889152,     'bwIn' => 0,     'bwOut' => 0,     'connected' => 0 }, 'cacheStats' => {     'cache' => {        'cache' => {            'sample_1.flv' => [                {                'coreId' => 2,                'useCount' => 25                }            ]        },        'cores' => {            '0' => 0,            '1' => 0,            '2' => 1        }    } }

Balancer_app SSAS application

This application consolidates statistics from all Edge servers by polling edge_app SSAS applications. Also, it functions as the primary entry point for all client requests for content playback; in other words, based on the collected statistics, it looks for a proper Edge server among the available servers.

For instance, you can enable load balancing based on just two criteria:

• Whether the content is in the file cache
• Number of active clients

The proper Edge server is found as follows: If the needed content is present in the file cache, the least-loaded fmscore process is found among all Edges where the content is present. Alternatively, if there is no content in the cache, then simply the least-loaded Edge server and fmscore process are found.

After the Edge server and fmscore process are found, the application must perform an RTMP redirect, returning a response to the client application containing the Edge server address and the ID of its fmscore process. The code would look like this:

application.onConnect = function(client, contentName) {    // find an edge server    var edge = this.findEdge(contentName);    // find core process id at it    var coreId = this.findCoreId(edge);    var errObj = {errorMsg: "Rejected due to redirection!", coreId: coreId};    application.redirectConnection(client, edge.srv.appURI, "Redirected!", errObj);}

Access plug-in

The Access plug-in must be installed on the Edge server to enable correct attachment of inbound connections based on an ID passed by the load balancer (SSAS balancer_app) to the fmscore process. For additional information, please refer to the beginning of this section or to Developing an Access plug-in in the Flash Media Server Developer's Guide.

Сlient application (player)

The client application (a Flash Player instance or an Adobe AIR application) should be able to interact with the load balancer (SSAS balancer_app): to make a request, interpret the response with RTMP redirect and properly construct the URI with the fmscore process ID (see Figure 6).

Figure 6. Interaction of the client application and the load balancer

First, the client application establishes a connection to the load balancer and provides to it the name of the content that the customer wants to play back:

var nc = new NetConnection();nc.connect(balancer_app_uri, "sample_1.flv");

In response, the load balancer finds a suitable Edge server and fmscore process, and returns an RTMP redirect (the command is sent to the user), specifying in the redirect parameters the Edge server URI and fmscore process ID. A client application, for example, can handle the response as follows:

public function onNCStatus(event:NetStatusEvent) : void {    if(event.info.code=="NetConnection.Connect.Rejected) {        var info:Object = event.info;        var app:Object = info.application;        if(info.ex && info.ex.code==302) {             var coreId = app.coreId);        }    }}

Finally, the client application must connect to the Edge server, passing the URI with the received fmscore process ID to the Edge server to be processed by the Access plug-in:

. . .var nc = new NetConnection(); var edgeURI = info.ex.redirect+"?coreId="+app.coreIdnc.connect(edgeURI); . . .

This is but one of the options to enable load balancing on a video platform. It represents quite a flexible approach to correct load balancing: for instance, to extend this example to live content balancing, it is sufficient to accumulate slightly different statistics on the Edge servers and ignore the file cache. Moreover, for DVD content, the solution would not require any change at all.

Where to go from here

For more information about developing plug-ins for Flash Media Server, try these resources:

• Large-scale streaming deployments with Flash Media Interactive Server 3.5
• Flash Media Server 4.5 Technical Overview (documentation)
• Flash Media Server 4.5 Developer's Guide (documentation)
• Developing plug-ins (documentation)
• Developing an Access plug-in (documentation)
• DENIVIP Media (blog)

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License