Best practices for real-time collaboration usin...

Best practices for real-time collaboration using Flash Media Server

by Jozsef Vass

• Adobe

Content

• Advantages of RTMFP for real-time collaborations
• Challenges of using RTMFP
• Using Flash Media Server for RTMFP communications
• Sample video meeting application
• Where to go from here

Was this helpful?

YesNo

0% out of 1 visitors found this helpful.

Created

13 December 2010

• Comments (0)

Page tools

Share on Facebook

Share on Twitter

Share on LinkedIn

Bookmark

Print

collaboration Flash Builder 4 Flash Media Server RTMFP

Requirements

Prerequisite knowledge

Familiarity with ActionScript 3 and Adobe Flash Builder.

User level

Advanced

Required products

• Flash Media Server (Download trial)
• Flash Builder 4 (Download trial)
• Flash Player 10 and later

Sample files

• rtmfp_app_assets.zip (20 KB) (18 KB)
• 
  

With the release of Flash Player 10, Adobe introduced Real-Time Media Flow Protocol (RTMFP). When used for real-time collaboration applications, RTMFP provides several advantages when compared to Real-Time Messaging Protocol (RTMP)—most notably, reduced latency and direct peer-to-peer connections.

To use RTMFP, however, Flash Player needs to connect to a real-time communications service such as that provided by Adobe Flash Media Server 4 or Codename Cirrus. The Cirrus service (rtmfp://p2p.rtmfp.net), hosted by Adobe, can be used for development purposes. You can view it as a stripped-down version of Flash Media Server (FMS) 4 that exclusively provides a rendezvous service: it does not support media relay, shared objects, remote method invocation, or other features of Flash Media Server 4.

To deploy commercial applications using RTMFP, you must use Flash Media Server 4. Not only can you take advantages of all FMS capabilities, but you can address firewall-related limitations of the Cirrus service that would prevent successful deployment of your application.

In this article, I first point out the advantages of RTMFP versus RTMP. I then describe the challenges faced by RTMFP. Next, I provide solutions for these challenges using Flash Media Server. Finally, I present a sample video meeting application demonstrating how to seamlessly work around different versions of Flash Player instances and network firewall devices.

This article is intended for intermediate and advanced ActionScript developers. It does not explain how to create basic streaming applications, but focuses on more advanced topics. For an introduction and API review of RTMFP, please read Cirrus service for developing end-to-end applications using RTMFP in Flash Player 10.

Adobe LiveCycle Collaboration Services (LCCS) is a high-level SDK that takes care of all the issues discussed here. This article is intended for developers who would like to get more control than that provided by LCCS.

With the release of Flash Player 10.1 and Flash Media Server 4, RTMFP further provides application-level and native IP multicast. These capabilities of RTMFP are out of the scope of the article.

Advantages of RTMFP for real-time collaborations

RTMP is an open protocol introduced in Flash Player 6 that mainly targets streaming audio and video content. Over time, several flavors of RTMP were developed, including tunneling over HTTP, providing protected content, and more. RTMP is based on Transmission Control Protocol (TCP). As such, it provides reliable data delivery between Flash Media Server and Flash Player, which is achieved at the price of unbounded delay. This is not a problem for broadcast-type media distribution (either live or recorded) because Flash Player maintains a playback buffer—typically several seconds long—that can compensate for network interruptions. In these applications, ensuring media integrity is of higher importance than timely delivery. Losing media data could also introduce undesirable audio/video distortion.

Flash Player 10 (and Adobe AIR 1.5) introduced RTMFP. Unlike RTMP, RTMFP is based on User Datagram Protocol (UDP). UDP provides fast and unreliable data delivery with minimal overhead; there is no flow control, congestion control, or other safeguards. UDP is popularly used for real-time communications applications where minimizing delay between communicating endpoints is of utmost importance. In addition, UDP also facilitates direct media exchange between communicating endpoints, which not only further reduces delay between endpoints, but also reduces media relay server requirements.

In real-time collaboration applications, minimizing delay is one of the most important goals; a few hundred milliseconds' worth of delay could render a conversation unusable. Achieving reliable transmission may not be needed since modern audio and video coding technologies (such as the Speex voice codec and the H.264 video codec, both available in Flash Player) are designed to deal with network transmission errors and thus provide error concealment techniques.

When compared to RTMP, RTMFP provides the following advantages for real-time collaborations applications:

• Low latency: Because it is built on top of UDP, RTMFP provides minimal latency, which is of paramount importance for real-time collaboration applications. Latency can be further reduced by using an end-to-end media delivery mechanism.
• Partial reliability: Depending on your application needs, RTMFP can provide reliable or unreliable transmission. Naturally, reliable transmission results in unbounded delay—the same as with RTMP. On the other hand, you can minimize latency by using unreliable transmission. RTMFP allows for sending media unreliably and data reliably over the same connection between two communicating endpoints. RTMFP performs congestion control over UDP for both reliable and unreliable transmissions.
• End-to-end media delivery: Media can be directly sent between Flash Player endpoints without routing through a central relay server, whereas RTMP requires all media sent through FMS. This not only reduces latency but also reduces server requirements. RTMFP assigns a peer identifier to each Flash Player endpoint. To receive media from a publishing Flash Player endpoint, a subscribing endpoint must specify the remote peer's ID. The publishing peer must explicitly approve each direct subscribing peer.
• Data prioritization: Audio, video, and data are transmitted with different priority in case of limited network resources. Audio is sent with higher priority than video, which is sent with higher priority than data (note, however, that data is always sent reliably).
• Encryption: Communications over RTMFP are always encrypted using AES 128-bit long keys, which are negotiated using the Diffie-Hellman key exchange method. It is important to note that RTMFP does not provide strong endpoint authentication methods such as SSL or RTMPS. However, Flash Player does expose secure nonces to application developers. These non-forgeable nonces are available to all communicating endpoints and are guaranteed to match, which can be used to protect against man-in-the-middle attacks.

Challenges of using RTMFP

To use Real-Time Media Flow Protocol (RTMFP), Flash Player needs to connect to a service (either the hosted Cirrus service or Flash Media Server). This is required to obtain a so-called peer identifier, a unique 256-bit random number that is used in establishing communications between peers.

When creating commercial applications, developers must address the following issues:

• UDP blocking: As I mentioned before, RTMFP is based on UDP. Many corporate firewalls block UDP traffic altogether. One of the workarounds is to configure a TURN proxy for RTMFP, but a more desirable solution is to fall back from RTMFP to RTMP or RTMP Tunneled (RTMPT).
• Convergence of RTMFP and RTMP: RTMFP cannot be used if Flash Player 9 or earlier is involved in collaboration, or if firewalls block UDP. It is important that Flash Player clients using either RTMP or RTMFP can exchange media with each other.
• Failover in case of firewall blocking: Even if both clients can successfully connect to the rendezvous service using RTMFP, there is no guarantee that clients can directly send media to each other. It really depends on the combination of firewalls located at clients to determine whether direct communication is possible. When a direct connection is not possible, failover to centrally relayed media is required.
• Multiparty communications: In a multiparty scenario, each client must send media to every other client. This requires symmetric uplink and downlink, which is a poor match to consumer ADSL or cable connections. When more than three communicating parties are involved, it is probably more advantageous to use client-server media delivery topology.
• User lookup: When connected to a rendezvous service, Flash Player creates a unique peer ID. In order to establish direct communications between two Flash Player endpoints, each must know the other's peer ID. It is the application's responsibility to exchange these peer IDs.

The following section reviews in detail how you can address each of these challenges by using Flash Media Server.

Using Flash Media Server for RTMFP communications

Flash Media Server (FMS) 4 can be used to build applications using all the advantages of RTMFP. In addition, one can take advantage of all existing features such as shared objects and remote method invocation (RMI).

The following subsections describe how to address each of the challenges of RTMFP described in the previous section, providing ActionScript source code for both client and server where applicable.

UDP blocking

While rarely the case with small office or home office (SOHO) firewalls, it is quite common for corporate firewalls to block UDP traffic altogether. One solution is to configure Flash Player to use a TURN proxy (Traversal Using Relays around NAT). As documented in the Adobe Flash Player 10.1 Administration Guide, Flash Player can be configured to support IETF Internet Draft "draft-ietf-behave-turn-08" without authentication by specifying the following option in mms.cfg:

RTMFPTURNProxy = ip_address_or_hostname_of_TURN_proxy

This solution requires your corporate IT department to develop and deploy a conforming TURN proxy.

A more desirable solution is to seamlessly fail over to an RTMP connection when an RTMFP connection to the server cannot be established. Unfortunately, there is no way to probe whether an RTMFP connection to the server will be successful. The only way is to attempt the connection and wait for a NetConnection.Connect.Success or NetConnection.Connect.Failed response. A failed connection will provide a NetConnection.Connect.Failed event after a 90-second timeout. One approach would be to try parallel connections of both RTMP and RTMFP to the server. Whenever RTMFP succeeds, the RTMP connection can simply be closed.

Another strategy is to try protocols sequentially in preference order (for instance, RTMFP, RTMP, RTMPT) with a given timeout. This sample client-side ActionScript code demonstrates sequential connection establishment:

private var rtmfpTimer:Timer = null; private var netConnection:NetConnection = null; private const ConnectionTimeout:int = 5000; public function register(rtmp:Boolean):void { netConnection = new NetConnection(); netConnection.addEventListener(NetStatusEvent.NET_STATUS, connectionHandler); netConnection.connect(rtmp ? "rtmp:" : "rtmfp:" + "//server.domain/application"); if (!rtmp) { rtmfpTimer = new Timer(ConnectionTimeout, 1); rtmfpTimer.addEventListener(TimerEvent.TIMER_COMPLETE, rtmfpTimeoutHandler); rtmfpTimer.start(); } } private function rtmfpTimeoutHandler(e:TimerEvent):void { netConnection.close(); netConnection = null; register(true); } private function connectionHandler(e:NetStatusEvent):void { if (rtmfpTimer) { rtmfpTimer.stop(); rtmfpTimer = null; } if ("NetConnection.Connect.Success" == e.info.code) { // RTMFP or RTMP connection succeeded } else if ("NetConnection.Connect.Failed" == e.info.code) { // RTMFP or RTMP connection failed } }

Convergence of RTMP and RTMFP

It is quite possible that not all Flash Player endpoints can use RTMFP. For instance:

• Flash Player 9 or an earlier version is used, and thus RTMFP is not available
• A firewall blocks UDP, and thus RTMFP is not possible

In both of these cases, the only solution is to send media through Flash Media Server. FMS will transparently bind RTMPx and RTMFP streams together: no server or client work is needed.

Since RTMP only works in a client-server scenario, your application is responsible for switching its conference topology between direct communications and client-server communications. Assume that there are two participants (say, A and B) in a video meeting using direct communications over RTMFP. When client C (using RTMP) joins, the meeting topology must be converted to client-server; that is, all media must go through the central FMS. When client C leaves, the meeting can be converted back to a direct connection. Note that clients A and B do not need to change protocol from RTMFP to RTMP; FMS will perform seamless bridging of the two protocols.

Failover in case of firewall blocking

Even if both Flash Player endpoints successfully connect to the RTMFP service, there is no guarantee that two clients can communicate directly with each other because intermediaries such as firewalls can block communications between two endpoints. RFC 3489, STUN—Simple Traversal of User Datagram Protocol (UDP) Through Network Address Translators—classified network address translators (NATs) into the following four categories: full cone, restricted cone, port restricted cone, and symmetric. (Full cone is the most permissive and symmetric is the most restrictive.) Later experimentation has found that firewalls cannot be clearly classified to the above categories; for instance, certain firewalls exhibit different behavior for incoming and outgoing traffic.

Although you can predict whether direct communications between two endpoints will be successful by knowing the firewall types, no guarantees can be given. The best way to see whether direct communication between two endpoints is possible is simply to try. This is the approach adopted by RFC 5245, Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols. First, a series of communication addresses are collected. These addresses include local addresses, external addresses, or relayed addresses. Communications starts with the address with the highest probability of success (which may not be the cheapest solution) and later changing to the cheapest possible solution (local address).

In this example, I've adopted a similar solution. To start, the application sends media both through the server and directly to peers. Sending media through FMS always succeeds. An alternative approach would be to simply probe using data messages to see whether direct connection between two clients is available. There are two ways to detect whether a direct connection was successful:

• Reception of NetStream.Play.Start on an incoming directly connected stream
• Reception of an onPeerConnect() callback on the NetStream client object on an outgoing directly connected stream

This following sample code demonstrates the idea. Here's the code for the publishing client:

private var outgoingStreamDirect:NetStream = null; private var outgoingStreamFms:NetStream = null; private function multiPublish():void { outgoingStreamDirect = new NetStream(netConnection, NetStream.DIRECT_CONNECTIONS); outgoingStreamFms = new NetStream(netConnection); outgoingStreamDirect.publish("streamDirect"); var c:Object = new Object(); c.onPeerConnect = function(peer:NetStream):Boolean { // if we receive onPeerConnect, it means that direct connection succeeded if (outgoingStreamFms) { outgoingStreamFms.close(); outgoingStreamFms = null; } return true; } outgoingStreamDirect.client = c; outgoingStreamFms.publish("streamFms") }

In the subscribing client, it is assumed that the publishing peer ID is already known:

private var incomingStreamDirect:NetStream = null; private var incomingStreamFms:NetStream = null; private function multiSubscribe():void { incomingStreamFms = new NetStream(netConnection); incomingStreamDirect = new NetStream(netConnection, id_of_publishing_client); incomingStreamDirect.addEventListener(NetStatusEvent.NET_STATUS, streamHandler); incomingStreamFms.play("streamFms"); incomingStreamDirect.play("streamDirect"); } private function streamHandler(e:NetStatusEvent):void { // if Play.Start event received on direct connection, it means that direct communications is possible if ("NetStream.Play.Start" == e.info.code) { if (incomingStreamFms) { incomingStreamFms.close(); incomingStreamFms = null; } } }

You have several strategies to choose from for when to start sending and receiving media. You may want to start sending and receiving media right away using the client-server stream, since that will succeed in every case. There may be a small glitch, however, when switching over from client-server to direct connections. Alternatively, you may want to wait to start with a direct connection and failover to client-server after a given timeout.

Multiparty communications

Direct connections are preferred between communicating clients to minimize latency and reduce server costs. In a multiparty session, each participating client must send media to all participating clients (see Figure 1). Thus, each client uses the same uplink and downlink bandwidth. Assuming each media stream requires b bandwidth, the required uplink or downlink bitrate is b (n–1), where n is the number of participating clients. This is a poor match for home users' cable or ADSL modems. Imagine a 500 Kbps video stream, even a few participants may easily overwhelm one's uplink, causing performance degradation. On the other hand, client-server topology is a perfect match for asymmetric network connections. It requires b uplink bandwidth and b (n–1) downlink.

Figure 1. Media transmission in direct connection (left) and client-server (right) topologies. Thin lines denote control connection and thick lines denote media data.

User lookup

RTMFP assigns a peer ID to each participant. These peer IDs are 256 bits long and are non-forgeable. When you want to subscribe to a directly published stream, you must specify the publisher's peer ID:

var receiveStream:NetStream = new NetStream(netConnection, id_of_publishing_client); receiveStream.play("media");

Therefore, your application needs to provide some kind of service to exchange peer IDs. You can use XMPP or a web service to exchange peer IDs. FMS offers remote procedure calls or remote shared objects to perform this task. The following example demonstrates how to exchange peer IDs using remote method invocation.

Here is the FMS Server-Side ActionScript (SSAS):

application.onConnect = function(client, userName) { client.userName = userName; // server-side functions client.getRemoteId = getRemoteId; application.acceptConnection(client); return true; } function findUser(userName) { for (var i = 0; i <= application.clients.length; i++) { if (application.clients[i].userName == userName) { return i; } } return -1; } function getRemoteId(userName) { var result = new Object; result.userName = userName; result.id = ""; result.protocol = ""; var index = findUser(userName); if (-1 != index) { result.id = application.clients[index].farID; result.protocol = application.clients[index].protocol; } return result; }

Here is the Flash Player code to register and look up the peer ID:

private var netConnection:NetConection = null; private function connect():void { netConnection = new NetConnection(); netConnection.connect("rtmfp://example.com/app", "user"); } private function lookup(remoteUser:String):void { var responder:Responder = new Responder(userResult); netConnection.call("getRemoteId", responder, remoteUser); } private function userResult(result:Object):void { // process respons // result.id; // result.userName; // result.protocol; }

Sample video meeting application

I have developed a sample application demonstrating the following items discussed in this article:

• Use the best connection method to the server. RTMFP is preferred, with fall-back to RTMP if RTMFP is not possible.
• Register users at the server, using remote method invocation for user lookup.
• Try to establish direct communications when there are three or fewer participants. If direct connection fails, use client-server data delivery.
• If there are more than three participants, switch topology to client-server. When a participant leaves, switch back to direct media delivery.
• If a communicating endpoint can only use RTMP for example, when UDP is blocked, use client-server conferencing topology.

The application was developed using Flash Builder 4. The minimum requirement is Flash Player 10; it does not handle Flash Player 9 gracefully (an update is required using Express Install). The application requires Flash Media Server 4. In order to minimize latency for real-time collaboration applications, disable Aggregate Messages and Queuing (by disabling AggregateMessages in Vhost.xml, Queue and AggregateMessages in Application.xml); see also Send aggregate messages in the Flash Media Server online documentation.

The following files are included:

• main.asc: Server-side code to manage meetings and send notifications and participant list when a user joins or leaves a meeting.
• VideoMeeting.mxml: Main application component, including user interface.
• ConnectionManager.as: Component handling the connection to FMS, with failover.
• SessionManager.as: Component handling participants and media; uses direct connection when possible with failover to client-server topology. Also includes outgoing NetStream object.
• LoginWindow.mxml: Login window component to provide FMS service information.
• Participant.as: Participant properties, including incoming NetStream object and corresponding Video display object.
• ParticipantListRenderer.mxml: Rendition of each participant in the main window user list.
• ParticipantEvent.as: Notification event when a participant joins or leaves the meeting.
• MessageEvent.as: Notification of incoming text messages.
• Settings.as: Application configuration properties (such as selected devices).
• Logger.as: Miscellaneous logging component.

Where to go from here

This article presented several aspects of developing real-time collaboration applications. After this article, you should have a good understanding on how to make the best use of RTMFP with Flash Media Server in your real-time communications apps.

Here are some resources to help you develop real-time communications apps for Flash Media Server and Flash Player clients:

• Cirrus service for developing end-to-end applications using RTMFP in Flash Player 10
• Peer-assisted networking using RTMFP groups in Flash Player 10.1
• Flash Media Server 4.0 Developer's Guide: RTMFP groups
• Open Source Media Framework Developer's Guide: Using RTMFP multicasting

This work is licensed under a Creative Commons Attribution-Noncommercial 3.0 Unported License.

More Like This

• Using peer-to-peer applications on the Adobe Flash Platform
• Using peer-to-peer applications on the Adobe Flash Platform
• Beginning Flash Media Server 4.5 – Part 5: Streaming with OSMF players

Tutorials & Samples

Tutorials

• Content protection using PHDS/PHLS and Adobe Access with Adobe Media Server 5.0
• SWF verification for Protected HTTP Dynamic Streaming
• Adobe Media Server 5 introduction white paper

Samples

• Best practices for real-time collaboration using Flash Media Server
• Understanding live DVR – Part 2: Using DVRCast with Flash Media Live Encoder 3
• Understanding live DVR – Part 1: Implementing a live DVR player

Flash Media Server Forum

More

05/07/2012	Flash on Google Chrome
05/08/2012	Republishing the Streams
05/08/2012	Publish multi-bitrate stream from FMS?
05/07/2012	How to change play video immediately ?

Comments (0)

Comments