Jitsi

 http://www.aosabook.org/en/jitsi.html

Jitsi is an application that allows people to make video and voicecalls, share their desktops, and exchange files and messages. Moreimportantly it allows people to do this over a number of differentprotocols, ranging from the standardized XMPP (Extensible Messagingand Presence Protocol) and SIP (Session Initiation Protocol) toproprietary ones like Yahoo! and Windows Live Messenger (MSN). Itruns on Microsoft Windows, Apple Mac OS X, Linux, and FreeBSD. It iswritten mostly in Java but it also contains parts written in nativecode. In this chapter, we'll look at Jitsi's OSGi-based architecture,see how it implements and manages protocols, and look back on whatwe've learned from building it.1

Jitsi是一个允许人们制作视频和语音的应用程序，分享他们的桌面，交换文件和信息。更重要的是，它允许人们通过许多不同的协议来实现这一点，从标准化的XMPP(可扩展的消息和到场协议)和SIP(会话发起协议)到像yahoo！还有Windows Live Messenger(MSN)。它运行在微软的Windows、苹果的Mac OS X、Linux和FreeBSD上。它主要是用Java编写的，但也包含在nativecode中编写的部分。在这一章中，我们将看到Jitsi的基于osgi的体系结构，看看它是如何实现和管理协议的，并回顾我们从构建它的过程中得到了什么。

10.1. Designing Jitsi

The three most important constraints that we had to keep in mind whendesigning Jitsi (at the time called SIP Communicator) weremulti-protocol support, cross-platform operation, anddeveloper-friendliness.

From a developer's perspective, being multi-protocol comes down tohaving a common interface for all protocols. In other words, when auser sends a message, our graphical user interface needs to alwayscall the same sendMessage method regardless of whether thecurrently selected protocol actually uses a method calledsendXmppMessage or sendSipMsg.

The fact that most of our code is written in Java satisfies, to alarge degree, our second constraint: cross-platform operation. Still,there are things that the Java Runtime Environment (JRE) does notsupport or does not do the way we'd like it to, such as capturingvideo from your webcam. Therefore, we need to use DirectShow onWindows, QTKit on Mac OS X, and Video for Linux 2 on Linux. Just aswith protocols, the parts of the code that control video calls cannotbe bothered with these details (they are complicated enough as it is).

Finally, being developer-friendly means that it should be easy forpeople to add new features. There are millions of people using VoIPtoday in thousands of different ways; various service providers andserver vendors come up with different use cases and ideas about newfeatures. We have to make sure that it is easy for them to use Jitsithe way they want. Someone who needs to add something new should haveto read and understand only those parts of the project they aremodifying or extending. Similarly, one person's changes should haveas little impact as possible on everyone else's work.

To sum up, we needed an environment where different parts of the codeare relatively independent from each other. It had to be possible toeasily replace some parts depending on the operating system; haveothers, like protocols, run in parallel and yet act the same; and ithad to be possible to completely rewrite any one of those parts andhave the rest of the code work without any changes. Finally, wewanted the ability to easily switch parts on and off, as well as theability to download plugins over the Internet to our list.

We briefly considered writing our own framework, but soon dropped theidea. We were itching to start writing VoIP and IM code as soon aspossible, and spending a couple of months on a plugin frameworkdidn't seem that exciting. Someone suggested OSGi, and it seemed to bethe perfect fit.

在设计Jitsi的时候，我们必须记住的三个最重要的约束(在称为SIP通信者)的时候，我们的协议支持、跨平台的操作和开发人员的友好性。

从开发人员的角度来看，多协议的出现是为了让所有协议都有一个通用的接口。换句话说，当auser发送一条消息时，我们的图形用户界面需要始终使用相同的sendMessage方法，而不管所选的协议是否实际使用了名为sendxmppmessage或sendSipMsg的方法。

我们的大多数代码都是用Java编写的，这就满足了我们的第二个约束条件:跨平台操作。尽管如此，Java运行时环境(JRE)并不支持或不支持我们所希望的方式，比如从您的网络摄像头中捕获视频。因此，我们需要在windows上使用DirectShow，在Mac OS X上使用QTKit，在Linux上使用Linux 2的视频。就像使用协议一样，控制视频通话的代码部分也不会被这些细节所困扰(因为它们已经足够复杂了)。

最后，开发人员友好意味着人们可以很容易地添加新特性。现在有成千上万的人以数千种不同的方式使用空洞;不同的服务提供者和服务器供应商提出了不同的用例和关于新特性的想法。我们必须确保他们能够很容易地使用他们想要的方式。那些需要添加新东西的人应该去阅读和理解那些他们激动或扩展的项目的部分。同样地，一个人的改变应该对其他人的工作产生尽可能少的影响。

总之，我们需要一个环境，在这个环境中，代码的不同部分彼此相对独立。它必须能够很容易地根据操作系统来替换某些部分;还有其他的，比如协议，并行运行，但却运行相同;并且它可以完全重写其中的任何一个部分，并且在没有任何变化的情况下使其余的代码工作。最后，我们希望能够轻松地切换各个部分，以及在互联网上下载插件的能力。

我们曾短暂地考虑过编写自己的框架，但很快就放弃了这个想法。我们渴望尽快开始编写VoIP和IM代码，并且花了几个月的时间在插件框架上，似乎并没有那么令人兴奋。有人推荐了OSGi，这似乎是完美的搭配。

10.2. Jitsi and the OSGi Framework

People have written entire books about OSGi, so we're not going to goover everything the framework stands for. Instead we will only explainwhat it gives us and the way we use it in Jitsi.

Above everything else, OSGi is about modules. Features in OSGiapplications are separated into bundles. An OSGi bundle is little morethan a regular JAR file like the ones used to distribute Javalibraries and applications. Jitsi is a collection of suchbundles. There is one responsible for connecting to Windows LiveMessenger, another one that does XMPP, yet another one that handlesthe GUI, and so on. All these bundles run together in an environmentprovided, in our case, by Apache Felix, an open source OSGiimplementation.

All these modules need to work together. The GUI bundle needs to sendmessages via the protocol bundles, which in turn need to store themvia the bundles handling message history. This is what OSGi servicesare for: they represent the part of a bundle that is visible toeveryone else. An OSGi service is most often a group of Javainterfaces that allow use of a specific functionality like logging,sending messages over the network, or retrieving the list of recentcalls. The classes that actually implement the functionality are knownas a service implementation. Most of them carry the name of theservice interface they implement, with an "Impl" suffix at the end(e.g., ConfigurationServiceImpl). The OSGi framework allowsdevelopers to hide service implementations and make sure that they arenever visible outside the bundle they are in. This way, other bundlescan only use them through the service interfaces.

Most bundles also have activators. Activators are simple interfacesthat define a start and a stop method. Every time Felixloads or removes a bundle in Jitsi, it calls these methods so that thebundle can prepare to run or shut down. When calling these methodsFelix passes them a parameter called BundleContext. The BundleContextgives bundles a way to connect to the OSGi environment. This way theycan discover whatever OSGi service they need to use, or register onethemselves (Figure 10.1).

人们已经写了关于OSGi的全部书籍，所以我们不会去讨论框架所代表的一切。相反，我们只会解释它给我们的是什么，以及我们在Jitsi中使用它的方式。

最重要的是，OSGi是关于模块的。OSGiapplications中的特性被分离成包。一个OSGi绑定包就像一个常规的JAR文件，就像用于分发Javalibraries和应用程序的JAR文件一样。Jitsi是这样的一群人的集合。有一个负责连接到Windows livemess格尔的人，另一个负责XMPP的人，还有另一个处理GUI的人，等等。在我们的例子中，所有这些包都是在一个环境中运行的，Apache Felix是一个开源的osgi实现。

所有这些模块都需要协同工作。GUI绑定需要通过协议束发送消息，而协议绑定则需要通过绑定消息历史的包来存储它们。这就是OSGi服务的用途:它们代表了对其他人可见的包的一部分。OSGi服务通常是一组Javainterfaces，它允许使用特定的功能，比如日志记录、通过网络发送消息或检索最近调用的列表。实际实现该功能的类被认为是一个服务实现。它们中的大多数都带有它们所实现的接口的名称，末尾有一个“Impl”后缀。ConfigurationServiceImpl)。OSGi框架允许开发人员隐藏服务实现，并确保在他们所处的包之外可以看到它们。通过这种方式，其他的bundles只能在服务接口中使用它们。

大多数包也有激活器。activator是一个简单的接口，它定义了一个开始和一个停止方法。每当费利克斯加载或删除Jitsi中的一个捆绑包时，它都会调用这些方法，使bundle能够准备运行或关闭。调用这些方法时，felix将传递一个名为BundleContext的参数。bundlecontext提供了一种连接到OSGi环境的方法。通过这种方式，他们可以发现需要使用的任何OSGi服务，或者注册一个自己的服务(图10.1)。

Figure 10.1: OSGi Bundle Activation

So let's see how this actually works. Imagine a service thatpersistently stores and retrieves properties. In Jitsi this is what wecall the ConfigurationService and it looks like this:

我们来看看它是如何工作的。想象一个持续存储和检索属性的服务。在Jitsi中这就是wecall的配置服务它是这样的:

package net.java.sip.communicator.service.configuration;

public interface ConfigurationService
{
  public void setProperty(String propertyName, Object property);
  public Object getProperty(String propertyName);
}

A very simple implementation of the ConfigurationService looks likethis:

package net.java.sip.communicator.impl.configuration;

import java.util.*;
import net.java.sip.communicator.service.configuration.*;

public class ConfigurationServiceImpl implements ConfigurationService
{
  private final Properties properties = new Properties();

  public Object getProperty(String name)
  {
    return properties.get(name);
  }

  public void setProperty(String name, Object value)
  {
    properties.setProperty(name, value.toString());
  }
}

Notice how the service is defined in thenet.java.sip.communicator.service package, while theimplementation is in net.java.sip.communicator.impl. Allservices and implementations in Jitsi are separated under these twopackages. OSGi allows bundles to only make some packages visibleoutside their own JAR, so the separation makes it easier for bundlesto only export their service packages and keep theirimplementations hidden.

The last thing we need to do so that people can start using ourimplementation is to register it in the BundleContext andindicate that it provides an implementation of theConfigurationService. Here's how this happens:

请注意在.java....net.service包中如何定义该服务，而实现则在net.java.sip.impl.impl中。在Jitsi中，所有的服务和实现都是在这两种情况下分离的。OSGi允许bundle只在自己的JAR之外进行一些包访问，所以分离使得bundles仅仅只能够导出它们的服务包并使其隐藏起来。

我们需要做的最后一件事是，人们可以开始使用使用的方法，就是在BundleContext中注册它，并表明它提供了这个功能服务的实现。这是如何发生的:

package net.java.sip.communicator.impl.configuration;

import org.osgi.framework.*;
import net.java.sip.communicator.service.configuration;

public class ConfigActivator implements BundleActivator
{
  public void start(BundleContext bc) throws Exception
  {
    bc.registerService(ConfigurationService.class.getName(), // service name
         new ConfigurationServiceImpl(), // service implementation
         null);
  }
}

Once the ConfigurationServiceImpl class is registered in theBundleContext, other bundles can start using it. Here's anexample showing how some random bundle can use our configurationservice:

package net.java.sip.communicator.plugin.randombundle;

import org.osgi.framework.*;
import net.java.sip.communicator.service.configuration.*;

public class RandomBundleActivator implements BundleActivator
{
  public void start(BundleContext bc) throws Exception
  {
    ServiceReference cRef = bc.getServiceReference(
                              ConfigurationService.class.getName());
    configService = (ConfigurationService) bc.getService(cRef);

    // And that's all! We have a reference to the service implementation
    // and we are ready to start saving properties:
    configService.setProperty("propertyName", "propertyValue");
  }
}

Once again, notice the package. Innet.java.sip.communicator.plugin we keep bundles that useservices defined by others but that neither export nor implement anythemselves. Configuration forms are a good example of such plugins:They are additions to the Jitsi user interface that allow users toconfigure certain aspects of the application. When users changepreferences, configuration forms interact with theConfigurationService or directly with the bundles responsiblefor a feature. However, none of the other bundles ever need tointeract with them in any way (Figure 10.2).

再一次，请注意这个包。在net.java.sip.a.plugin.插件中，我们保留了由其他人定义的服务包，但它们既不导出也不实现任何自己。配置表单是这些插件的一个很好的例子:它们是Jitsi用户界面的附加部分，允许用户配置应用程序的某些方面。当用户改变首选项时，配置表单会与这个功能服务交互，或者直接与绑定包负责一个特性。然而，其他任何bundle都不需要以任何方式与它们交互(图10.2)。

Figure 10.2: Service Structure

10.3. Building and Running a Bundle

Now that we've seen how to write the code in a bundle, it's time totalk about packaging. When running, all bundles need to indicate threedifferent things to the OSGi environment: the Java packages they makeavailable to others (i.e. exported packages), the ones that they wouldlike to use from others (i.e. imported packages), and the name oftheir BundleActivator class. Bundles do this through the manifest ofthe JAR file that they will be deployed in.

For the ConfigurationService that we defined above, themanifest file could look like this:

现在我们已经了解了如何在一个包中编写代码，现在是讨论打包的时候了。在运行时，所有的包都需要向OSGi环境显示三种不同的东西:它们可以向其他人提供的Java包(即导出的包)，它们想要从其他的包中使用的Java包(即导入的包)，以及它们的bunactivactivator类的名称。捆绑包通过JAR文件的清单来实现，这些文件将被部署到其中。

对于我们在上面定义的配置服务，themanifest文件可能是这样的:

Bundle-Activator: net.java.sip.communicator.impl.configuration.ConfigActivator
Bundle-Name: Configuration Service Implementation
Bundle-Description: A bundle that offers configuration utilities
Bundle-Vendor: jitsi.org
Bundle-Version: 0.0.1
System-Bundle: yes
Import-Package: org.osgi.framework,
Export-Package: net.java.sip.communicator.service.configuration

After creating the JAR manifest, we are ready to create the bundleitself. In Jitsi we use Apache Ant to handle all build-relatedtasks. In order to add a bundle to the Jitsi build process, you needto edit the build.xml file in the root directory of theproject. Bundle JARs are created at the bottom of thebuild.xml file, with bundle-xxx targets. In order tobuild our configuration service we need the following:

As you can see, the Ant target simply creates a JAR file using ourconfiguration manifest, and adds to it the configuration packages fromthe service and impl hierarchies. Now the only thingthat we need to do is to make Felix load it.

We already mentioned that Jitsi is merely a collection of OSGibundles. When a user executes the application, they actually startFelix with a list of bundles that it needs to load. You can find thatlist in our lib directory, inside a file calledfelix.client.run.properties. Felix starts bundles in the orderdefined by start levels: All those within a particular level areguaranteed to complete before bundles in subsequent levels startloading. Although you can't see this in the example code above, ourconfiguration service stores properties in files so it needs to useourFileAccessService, shipped within the fileaccess.jarfile. We'll therefore make sure that the ConfigurationService startsafter the FileAccessService:

正如您所看到的，Ant目标只是使用我们的配置清单创建一个JAR文件，并向它添加来自服务和impl层次结构的配置包。现在我们要做的唯一一件事就是让Felix加载它。
我们已经提到过，Jitsi只是一个osgi捆的集合。当用户执行该应用程序时，他们实际上会启动felix的列表，其中需要加载的包列表。您可以在我们的lib目录中找到这个列表，它位于一个名为“费利”.client.run.properties的文件中。Felix从开始级别定义的顺序开始:在一个特定的级别中，所有这些都保证在随后的级别开始加载之前完成。尽管您在上面的示例代码中看不到这一点，但是我们的配置服务将属性存储在文件中，因此它需要使用文件accessservice，在fileaccess.jarfile中发送。因此，我们将确保配置服务在FileAccessService之后启动:
⋮    ⋮    ⋮
felix.auto.start.30= \
  reference:file:sc-bundles/fileaccess.jar

felix.auto.start.40= \
  reference:file:sc-bundles/configuration.jar \
  reference:file:sc-bundles/jmdnslib.jar \
  reference:file:sc-bundles/provdisc.jar \
⋮    ⋮    ⋮

If you look at the felix.client.run.properties file, you'll seea list of packages at the beginning:

org.osgi.framework.system.packages.extra= \
  apple.awt; \
  com.apple.cocoa.application; \
  com.apple.cocoa.foundation; \
  com.apple.eawt; \
⋮    ⋮    ⋮

The list tells Felix what packages it needs to make available tobundles from the system classpath. This means that packages that areon this list can be imported by bundles (i.e. added to theirImport-Package manifest header) without any being exported byany other bundle. The list mostly contains packages that come fromOS-specific JRE parts, and Jitsi developers rarely need to add newones to it; in most cases packages are made available by bundles.

该列表告诉Felix需要从系统类路径中获得哪些包。这意味着该列表上的包可以通过捆绑包(即添加到irimport-package清单头)导入，而不需要任何其他bundle导出。该列表主要包含来自特定于os的JRE部分的包，而Jitsi开发人员很少需要向它添加新的包;在大多数情况下，包是由包提供的。

10.4. Protocol Provider Service

The ProtocolProviderService in Jitsi defines the way allprotocol implementations behave. It is the interface that otherbundles (like the user interface) use when they need to send andreceive messages, make calls, and share files through the networksthat Jitsi connects to.

The protocol service interfaces can all be found under thenet.java.sip.communicator.service.protocol package. There aremultiple implementations of the service, one per supported protocol,and all are stored innet.java.sip.communicator.impl.protocol.protocol_name.

Let's start with the service.protocol directory. The mostprominent piece is the ProtocolProviderService interface.Whenever someone needs to perform a protocol-related task, they haveto look up an implementation of that service in theBundleContext. The service and its implementations allow Jitsito connect to any of the supported networks, to retrieve the connectionstatus and details, and most importantly to obtain references to theclasses that implement the actual communications tasks like chattingand making calls.

10.4.1. Operation Sets

As we mentioned earlier, the ProtocolProviderService needs toleverage the various communication protocols and theirdifferences. While this is particularly simple for features that allprotocols share, like sending a message, things get trickier for tasksthat only some protocols support. Sometimes these differences comefrom the service itself: For example, most of the SIP services outthere do not support server-stored contact lists, while this is arelatively well-supported feature with all other protocols. MSN andAIM are another good example: at one time neither of them offered theability to send messages to offline users, while everyone elsedid. (This has since changed.)

The bottom line is our ProtocolProviderService needs to have away of handling these differences so that other bundles, like the GUI,act accordingly; there's no point in adding a call button to an AIMcontact if there's no way to actually make a call.

OperationSets to the rescue(Figure 10.3). Unsurprisingly, they are sets ofoperations, and provide the interface that Jitsi bundles use tocontrol the protocol implementations. The methods that you find in anoperation set interface are all related to a particular feature.OperationSetBasicInstantMessaging, for instance, containsmethods for creating and sending instant messages, and registeringlisteners that allow Jitsi to retrieve messages it receives. Anotherexample, OperationSetPresence, has methods for querying thestatus of the contacts on your list and setting a status foryourself. So when the GUI updates the status it shows for a contact,or sends a message to a contact, it is first able to ask thecorresponding provider whether they support presence andmessaging. The methods that ProtocolProviderService defines forthat purpose are:

public Map getSupportedOperationSets();
public  T getOperationSet(Class opsetClass);

OperationSets have to be designed so that it is unlikely that a newprotocol we add has support for only some of the operations defined inan OperationSet. For example, some protocols do not support server-storedcontact lists even though they allow users to query each other's status.Therefore, rather than combining the presence management and buddy listretrieval features in OperationSetPresence, we also defined anOperationSetPersistentPresence which is only used with protocolsthat can store contacts online. On the other hand, we have yet to comeacross a protocol that only allows sending messages without receivingany, which is why things like sending and receiving messages can besafely combined.

Jitsi中的协议providerservice定义了所有协议实现的方式。当他们需要发送和接收消息、打电话和通过Jitsi连接的网络共享文件时，其他的界面(比如用户界面)就会使用。

协议服务接口可以在下面的.java.sip.service.service.protocol包中找到。该服务的实现有一个支持协议，一个支持协议，所有的服务都被存储在net.java.sip...net.协议。协议名中。

让我们从服务开始。协议目录。最突出的部分是协议providerservice接口。每当有人需要执行与协议相关的任务时，他们就会在bundlecontext中查找该服务的实现。这项服务和它的实现允许Jitsito连接到任何受支持的网络，以检索连接状态和细节，最重要的是获得对实现实际通信任务的引用，比如聊天和打电话。

10.4.1。操作集

正如我们前面提到的，协议providerservice需要对各种通信协议及其差异进行详细的介绍。虽然这对于所有协议共享的特性来说都特别简单，比如发送消息，但是对于只有一些协议支持的任务来说，事情变得更加棘手。有时，这些差异来自服务本身:例如，大多数的SIP服务都不支持服务器存储的联系人列表，而这是与所有其他协议一起支持的功能良好的特性。MSN和aim是另一个很好的例子:在同一时间，他们都没有提供向离线用户发送消息的能力，而其他所有人都是这样做的。(这已经发生了改变。)

底线是我们的协议提供程序服务需要处理这些差异，这样其他bundle，比如GUI，就会相应地采取行动;如果没有办法进行调用，那么向目标用户添加一个呼叫按钮是没有意义的。

操作集合到救援(图10.3)。意料之中的是，它们是一组操作，并提供了Jitsi捆绑的接口来控制协议的实现。您在一个操作集接口中找到的方法都与一个特定的特性相关。例如，operationsetbasicinstant短信，包括创建和发送即时消息的方法，以及注册监听器，允许Jitsi检索它收到的消息。另一个例子是operationset风度，它有方法查询列表中的联系人的状态，并为自己设置一个状态。因此，当GUI更新它显示给联系人的状态，或者向联系人发送消息时，它首先能够询问响应的提供者是否支持存在和消息传递。协议提供服务定义的方法是:

公共Map < String,OperationSet > getSupportedOperationSets();

公共的作用集(类的作用:opsetClass);

操作集必须被设计，因此我们添加的一个新协议不太可能支持定义inan操作集的一些操作。例如，有些协议不支持服务器存储联系人列表，即使它们允许用户查询彼此的状态。因此，我们并没有将存在管理和伙伴listretrieval的功能组合在operationset到场中，而是定义了一个仅用于在网上存储联系人的协议。另一方面，我们还没有遇到一个协议，它只允许发送消息而不接收任何消息，这就是为什么发送和接收消息可以被合并的原因。

Figure 10.3: Operation Sets

10.4.2. Accounts, Factories and Provider Instances

An important characteristic of the ProtocolProviderService isthat one instance corresponds to one protocol account. Therefore, atany given time you have as many service implementations in theBundleContext as you have accounts registered by the user.

At this point you may be wondering who creates and registers theprotocol providers. There are two different entities involved. First,there is ProtocolProviderFactory. This is the service thatallows other bundles to instantiate providers and then registers themas services. There is one factory per protocol and every factory isresponsible for creating providers for that particularprotocol. Factory implementations are stored with the rest of theprotocol internals. For SIP, for example we havenet.java.sip.communicator.impl.protocol.sip.ProtocolProviderFactorySipImpl.

The second entity involved in account creation is the protocol wizard.Unlike factories, wizards are separated from the rest of the protocolimplementation because they involve the graphical user interface. Thewizard that allows users to create SIP accounts, for example, can befound in net.java.sip.communicator.plugin.sipaccregwizz.

ProtocolProviderFactory。这个服务允许其他bundle实例化提供者，然后将它们注册为服务。每个协议都有一个工厂，每个工厂都负责为特定的协议创建提供者。工厂实现与协议内部的其他部分一起存储。对于SIP，例如我们的..java..........协议.........................。

创建帐户创建的第二个实体是协议向导。与工厂不同的是，向导与其他的原始代码是分离的，因为它们涉及图形用户界面。例如，允许用户创建SIP账户的向导可以在net.java.sip.ac.plugin.sipaccregwizz中找到。

10.5. Media Service

When working with real-time communication over IP, there is oneimportant thing to understand: protocols like SIP and XMPP, whilerecognized by many as the most common VoIP protocols, are not the onesthat actually move voice and video over the Internet. This task ishandled by the Real-time Transport Protocol (RTP). SIP and XMPP areonly responsible for preparing everything that RTP needs, likedetermining the address where RTP packets need to be sent andnegotiating the format that audio and video need to be encoded in(i.e. codec), etc. They also take care of things like locating users,maintaining their presence, making the phones ring, and manyothers. This is why protocols like SIP and XMPP are often referred toas signalling protocols.

What does this mean in the context of Jitsi? Well, first of all itmeans that you are not going to find any code manipulating audio orvideo flows in either the sip or jabber jitsi packages.This kind of code lives in our MediaService. The MediaService and itsimplementation are located innet.java.sip.communicator.service.neomedia andnet.java.sip.communicator.impl.neomedia.

Why "neomedia"?

The "neo" in the neomedia package name indicates that it replaces asimilar package that we used originally and that we then had tocompletely rewrite. This is actually how we came up with one of ourrules of thumb: It is hardly ever worth it to spend a lot of timedesigning an application to be 100% future-proof. There is simply noway of taking everything into account, so you are bound to have tomake changes later anyway. Besides, it is quite likely that apainstaking design phase will introduce complexities that you willnever need because the scenarios you prepared for never happen.

In addition to the MediaService itself, there are two other interfacesthat are particularly important: MediaDevice and MediaStream.

10.5.1. Capture, Streaming, and Playback

MediaDevices represent the capture and playback devices that we useduring a call (Figure 10.4). Your microphone andspeakers, your headset and your webcam are all examples of suchMediaDevices, but they are not the only ones. Desktop streaming andsharing calls in Jitsi capture video from your desktop, while aconference call uses an AudioMixer device in order to mix the audio wereceive from the active participants. In all cases, MediaDevicesrepresent only a single MediaType. That is, they can only be eitheraudio or video but never both. This means that if, for example, youhave a webcam with an integrated microphone, Jitsi sees it as twodevices: one that can only capture video, and another one that canonly capture sound.

Devices alone, however, are not enough to make a phone or a videocall. In addition to playing and capturing media, one has to also beable to send it over the network. This is where MediaStreams comein. A MediaStream interface is what connects a MediaDevice to yourinterlocutor. It represents incoming and outgoing packets that youexchange with them within a call.

Just as with devices, one stream can be responsible for only oneMediaType. This means that in the case of an audio/video call Jitsihas to create two separate media streams and then connect each to thecorresponding audio or video MediaDevice.

当处理IP上的实时通信时，有一件重要的事情需要理解:像SIP和XMPP这样的协议，被许多人看作是最常见的VoIP协议，并不是真正在Internet上移动语音和视频的。这个任务是由实时传输协议(RTP)完成的。SIP和XMPP只负责准备RTP需要的所有内容，比如确定需要发送RTP包的地址，并协商音频和视频需要编码的格式(即:codec)等等。他们还负责定位用户，维护他们的存在，打电话，以及其他许多人。这就是为什么SIP和XMPP这样的协议通常被称为信号协议。

这在Jitsi的背景下意味着什么?首先，这意味着您将不会在sip或jabber jitsi包中找到任何操纵音频或视频流的代码。这种代码存在于我们的MediaService中。MediaService和它的服务定位于网络.java...................................。

为什么“neomedia”?

neomedia包中的“neo”表示它取代了我们最初使用的asimilar包，然后我们不得不完全重写。这实际上是我们的一个经验法则:花大量时间来设计一个应用程序以保证100%的未来是不值得的。现在很简单的把所有的事情都考虑进去，所以无论如何你都必须要做些改变。此外，很有可能的是，apain下注设计阶段会引入你永远不需要的复杂性，因为你准备的场景永远不会发生。

除了MediaService本身之外，还有另外两种特别重要的接口:media设备和MediaStream。

10.5.1。捕获、流媒体和回放

media设备代表了我们使用调用的捕获和回放设备(图10.4)。你的麦克风和扬声器，你的耳机和摄像头都是这些媒体设备的例子，但它们不是唯一的。在Jitsi中，桌面流媒体和分享呼叫可以从你的桌面获取视频，而一个会议调用使用AudioMixer设备来混合来自活跃参与者的音频。在所有情况下，media设备只表示一个MediaType。也就是说，它们只能是音频或视频，但两者都不能。这意味着，如果你有一个带有集成麦克风的网络摄像头，Jitsi将它视为两种设备:一种只能捕捉视频，另一种只能捕捉声音。

然而，仅靠设备是不够的，不足以制造一部电话或一部视频。除了播放和获取媒体，一个人还必须能够通过网络发送它。这就是mediast成堆的地方。MediaStream接口是将一个media设备连接到你的对话者的东西。它表示传入和传出的数据包，并在一个调用中与它们交换。

就像设备一样，一个流可以只负责oneMediaType。这意味着，在一个音频/视频通话的情况下，jitsi不得不创建两个独立的媒体流，然后将它们连接到响应的音频或视频媒体设备。

Figure 10.4: Media Streams For Different Devices

10.5.2. Codecs

Another important concept in media streaming is that of MediaFormats,also known as codecs. By default most operating systems let youcapture audio in 48KHz PCM or something similar. This is what weoften refer to as "raw audio" and it's the kind of audio you get inWAV files: great quality and enormous size. It is quite impractical totry and transport audio over the Internet in the PCM format.

This is what codecs are for: they let you present and transport audioor video in a variety of different ways. Some audio codecs like iLBC,8KHz Speex, or G.729, have low bandwidth requirements but soundsomewhat muffled. Others like wideband Speex and G.722 give you greataudio quality but also require more bandwidth. There are codecs thattry to deliver good quality while keeping bandwidth requirements at areasonable level. H.264, the popular video codec, is a good exampleof that. The trade-off here is the amount of calculation requiredduring conversion. If you use Jitsi for an H.264 video call you see agood quality image and your bandwidth requirements are quitereasonable, but your CPU runs at maximum.

All this is an oversimplification, but the idea is that codec choiceis all about compromises. You either sacrifice bandwidth, quality, CPUintensity, or some combination of those. People working with VoIPrarely need to know more about codecs.

10.5.3. Connecting with the Protocol Providers

Protocols in Jitsi that currently have audio/video support all use ourMediaServices exactly the same way. First they ask the MediaServiceabout the devices that are available on the system:

public List getDevices(MediaType mediaType, MediaUseCase useCase);

The MediaType indicates whether we are interested in audio or videodevices. The MediaUseCase parameter is currently only considered inthe case of video devices. It tells the media service whether we'dlike to get devices that could be used in a regular call(MediaUseCase.CALL), in which case it returns a list of availablewebcams, or a desktop sharing session (MediaUseCase.DESKTOP), in whichcase it returns references to the user desktops.

The next step is to obtain the list of formats that are available fora specific device. We do this through theMediaDevice.getSupportedFormatsmethod:

public List getSupportedFormats();

Once it has this list, the protocol implementation sends it to theremote party, which responds with a subset of them to indicate whichones it supports. This exchange is also known as the Offer/AnswerModel and it often uses the Session Description Protocol or some formof it.

After exchanging formats and some port numbers and IP addresses, VoIPprotocols create, configure and start the MediaStreams. Roughlyspeaking, this initialization is along the following lines:

// first create a stream connector telling the media service what sockets
// to use when transport media with RTP and flow control and statistics
// messages with RTCP
StreamConnector connector =  new DefaultStreamConnector(rtpSocket, rtcpSocket);
MediaStream stream = mediaService.createMediaStream(connector, device, control);

// A MediaStreamTarget indicates the address and ports where our
// interlocutor is expecting media. Different VoIP protocols have their
// own ways of exchanging this information
stream.setTarget(target);

// The MediaDirection parameter tells the stream whether it is going to be
// incoming, outgoing or both
stream.setDirection(direction);

// Then we set the stream format. We use the one that came
// first in the list returned in the session negotiation answer.
stream.setFormat(format);

// Finally, we are ready to actually start grabbing media from our
// media device and streaming it over the Internet
stream.start();

Now you can wave at your webcam, grab the mic and say, "Helloworld!"

公共列表< MediaFormat > getSupportedFormats();

一旦它有了这个列表，协议实现将它发送到theremote party，它会用一个子集来响应它，以表明它支持哪一个。这个交换也被称为off/应答模型，它经常使用会话描述协议或某种形式的协议。

在交换格式和一些端口号和IP地址之后，voip协议创建、配置和启动mediast成堆。粗略地说，这个初始化是沿着以下几行:

//首先创建一个流连接器，告诉媒体服务什么插座

//在运输媒体与RTP和流量控制和统计数据时使用

/ /消息服务器

StreamConnector连接器=new DefaultStreamConnector(rtpSocket，rtcpSocket);

= mediaService MediaStream流。createMediaStream(连接器、设备、控制);

//一个MediaStreamTarget显示我们的地址和端口

//对话者正在期待媒体的报道。不同的VoIP协议都有

//自己的交换信息的方式

stream.setTarget(目标);

//媒体方向参数告诉流，它是否将会

//进来，外向，或两者兼而有之

stream.setDirection(方向);

然后我们设置了流格式。我们用的是那个来的

//首先在会话协商答案中返回的列表中。

stream.setFormat(格式);

//最后，我们准备好开始从我们这里获取媒体

//媒体设备，并在互联网上播放

stream.start();

现在你可以在你的网络摄像头前挥一挥，拿起麦克风说:“Helloworld！”

10.6. UI Service

So far we have covered parts of Jitsi that deal with protocols,sending and receiving messages and making calls. Above all, however,Jitsi is an application used by actual people and as such, one of itsmost important aspects is its user interface. Most of the time theuser interface uses the services that all the other bundles in Jitsiexpose. There are some cases, however, where things happen the otherway around.

Plugins are the first example that comes to mind. Plugins in Jitsioften need to be able to interact with the user. This means they haveto open, close, move or add components to existing windows and panelsin the user interface. This is where our UIService comes into play. Itallows for basic control over the main window in Jitsi and this is howour icons in the Mac OS X dock and the Windows notification area letusers control the application.

In addition to simply playing with the contact list, plugins can alsoextend it. The plugin that implements support for chat encryption(OTR) in Jitsi is a good example for this. Our OTR bundle needs toregister several GUI components in various parts of the userinterface. It adds a padlock button in the chat window and asub-section in the right-click menu of all contacts.

The good news is that it can do all this with just a few method calls.The OSGi activator for the OTR bundle, OtrActivator, contains thefollowing lines:

除了简单地使用联系人列表之外，插件还可以扩展它。在Jitsi中实现对聊天加密(OTR)的支持的插件就是一个很好的例子。我们的OTR包需要在用户界面的不同部分中使用多个GUI组件。它在聊天窗口中添加了一个挂锁按钮，并在所有联系人的右键菜单中添加了一个按钮。

好消息是，它可以通过一些方法调用来完成所有这些工作。OTR包的OSGi激活器，OtrActivator，包含以下行

Hashtable filter = new Hashtable();

// Register the right-click menu item.
filter(Container.CONTAINER_ID,
    Container.CONTAINER_CONTACT_RIGHT_BUTTON_MENU.getID());

bundleContext.registerService(PluginComponent.class.getName(),
    new OtrMetaContactMenu(Container.CONTAINER_CONTACT_RIGHT_BUTTON_MENU),
    filter);

// Register the chat window menu bar item.
filter.put(Container.CONTAINER_ID,
           Container.CONTAINER_CHAT_MENU_BAR.getID());

bundleContext.registerService(PluginComponent.class.getName(),
           new OtrMetaContactMenu(Container.CONTAINER_CHAT_MENU_BAR),
           filter);

As you can see, adding components to our graphical user interfacesimply comes down to registering OSGi services. On the other side ofthe fence, our UIService implementation is looking for implementationsof its PluginComponent interface. Whenever it detects that a newimplementation has been registered, it obtains a reference to it andadds it to the container indicated in the OSGi service filter.

Here's how this happens in the case of the right-click menuitem. Within the UI bundle, the class that represents the right clickmenu, MetaContactRightButtonMenu, contains the following lines:

// Search for plugin components registered through the OSGI bundle context.
ServiceReference[] serRefs = null;

String osgiFilter = "("
    + Container.CONTAINER_ID
    + "="+Container.CONTAINER_CONTACT_RIGHT_BUTTON_MENU.getID()+")";

serRefs = GuiActivator.bundleContext.getServiceReferences(
        PluginComponent.class.getName(),
        osgiFilter);
// Go through all the plugins we found and add them to the menu.
for (int i = 0; i < serRefs.length; i ++)
{
    PluginComponent component = (PluginComponent) GuiActivator
        .bundleContext.getService(serRefs[i]);

    component.setCurrentContact(metaContact);

    if (component.getComponent() == null)
        continue;

    this.add((Component)component.getComponent());
}

And that's all there is to it. Most of the windows that you see withinJitsi do exactly the same thing: They look through the bundle contextfor services implementing the PluginComponent interface that have afilter indicating that they want to be added to the correspondingcontainer. Plugins are like hitch-hikers holding up signs with thenames of their destinations, making Jitsi windows the drivers who pickthem up.

10.7. Lessons Learned

When we started work on SIP Communicator, one of the most commoncriticisms or questions we heard was: "Why are you using Java? Don'tyou know it's slow? You'd never be able to get decent quality foraudio/video calls!" The "Java is slow" myth has even been repeatedby potential users as a reason they stick with Skype instead of tryingJitsi. But the first lesson we've learned from our work on the projectis that efficiency is no more of a concern with Java than it wouldhave been with C++ or other native alternatives.

We won't pretend that the decision to choose Java was the result ofrigorous analysis of all possible options. We simply wanted an easyway to build something that ran on Windows and Linux, and Java and theJava Media Framework seemed to offer one relatively easy way of doingso.

Throughout the years we haven't had many reasons to regret thisdecision. Quite the contrary: even though it doesn't make itcompletely transparent, Java does help portability and 90% of the codein SIP Communicator doesn't change from one OS to the next. Thisincludes all the protocol stack implementations (e.g., SIP, XMPP, RTP,etc.) that are complex enough as they are. Not having to worry aboutOS specifics in such parts of the code has proven immensely useful.

Furthermore, Java's popularity has turned out to be very importantwhen building our community. Contributors are a scarce resource as itis. People need to like the nature of the application, they need tofind time and motivation—all of this is hard to muster. Notrequiring them to learn a new language is, therefore, an advantage.

Contrary to most expectations, Java's presumed lack of speed hasrarely been a reason to go native. Most of the time decisions to usenative languages were driven by OS integration and how much accessJava was giving us to OS-specific utilities. Below we discussthe three most important areas where Java fell short.

当我们开始研究SIP通信时，我们听到的最常见的批评或问题之一是:“为什么要使用Java?”不是吗知道它是慢?你永远都无法获得高质量的音频/视频通话！“Java是缓慢的”这个神话甚至被潜在的用户反复使用，这是他们坚持使用Skype而不是使用trjitsi的原因。但是，我们从我们的项目中学到的第一个教训是，效率不再是与Java有关的问题，而不是C++或其他原生的替代方案。

我们不会假装选择Java的决定是对所有可能选项的严格分析的结果。我们只是想要一个简单的方法来构建在Windows和Linux上运行的东西，而Java和Java媒体框架似乎提供了一种相对简单的方法。

这些年来，我们没有太多理由对这个决定感到后悔。恰恰相反:尽管它并没有完全透明，但Java确实帮助了可移植性，并且90%的codesip通信器不会从一个操作系统转到另一个操作系统。这包括所有协议栈实现(例如:SIP、XMPP、RTP等)它们都是非常复杂的。不必担心代码中这些部分的细节已经被证明是非常有用的。

此外，Java的流行在构建我们的社区时是非常重要的。贡献者是一种稀缺资源。人们需要喜欢应用程序的本质，他们需要找到时间和动机——所有这些都是很难找到的。因此，要求他们学习一门新语言是一种优势。

与大多数人的预期相反，Java认为缺乏速度是不可能成为原生的原因。大多数时候，对使用语言的决定都是由操作系统的集成驱动的，并且有多少accessJava给我们提供了特定于OS的实用工具。下面我们讨论了Java失败的三个最重要的领域。

10.7.1. Java Sound vs. PortAudio

Java Sound is Java's default API for capturing and playing audio. Itis part of the runtime environment and therefore runs on all theplatforms the Java Virtual Machine comes for. During its first yearsas SIP Communicator, Jitsi used JavaSound exclusively and thispresented us with quite a few inconveniences.

First of all, the API did not give us the option of choosing whichaudio device to use. This is a big problem. When using their computerfor audio and video calls, users often use advanced USB headsets orother audio devices to get the best possible quality. When multipledevices are present on a computer, JavaSound routes all audio throughwhichever device the OS considers default, and this is not good enoughin many cases. Many users like to keep all other applications runningon their default sound card so that, for example, they could keephearing music through their speakers. What's even more important isthat in many cases it is best for SIP Communicator to send audionotifications to one device and the actual call audio to another,allowing a user to hear an incoming call alert on their speakers evenif they are not in front of the computer and then, after picking upthe call, to start using a headset.

None of this is possible with Java Sound. What's more, the Linuximplementation uses OSS which is deprecated on most of today's Linuxdistributions.

We decided to use an alternative audio system. We didn't want tocompromise our multi-platform nature and, if possible, we wanted toavoid having to handle it all by ourselves. This is wherePortAudio2 came in extremely handy.

When Java doesn't let you do something itself, cross-platform opensource projects are the next best thing. Switching to PortAudio hasallowed us to implement support for fine-grained configurable audiorendering and capture just as we described it above. It also runs onWindows, Linux, Mac OS X, FreeBSD and others that we haven't had thetime to provide packages for.

10.7.2. Video Capture and Rendering

Video is just as important to us as audio. However, this didn't seemto be the case for the creators of Java, because there is no defaultAPI in the JRE that allows capturing or rendering video. For a whilethe Java Media Framework seemed to be destined to become such an APIuntil Sun stopped maintaining it.

Naturally we started looking for a PortAudio-style video alternative,but this time we weren't so lucky. At first we decided to go with theLTI-CIVIL framework from KenLarson3. This is a wonderfulproject and we used it for quite a while4. However it turned out to be suboptimalwhen used in a real-time communications context.

So we came to the conclusion that the only way to provide impeccablevideo communication for Jitsi would be for us to implement nativegrabbers and renderers all by ourselves. This was not an easy decisionsince it implied adding a lot of complexity and a substantialmaintenance load to the project but we simply had no choice: we reallywanted to have quality video calls. And now we do!

Our native grabbers and renderers directly use Video4Linux 2, QTKitand DirectShow/Direct3D on Linux, Mac OS X, and Windows respectively.

10.7.3. Video Encoding and Decoding

SIP Communicator, and hence Jitsi, supported video calls from itsfirst days. That's because the Java Media Framework allowed encodingvideo using the H.263 codec and a 176x144 (CIF) format. Those of youwho know what H.263 CIF looks like are probably smiling right now; fewof us would use a video chat application today if that's all it had tooffer.

In order to offer decent quality we've had to use other libraries likeFFmpeg. Video encoding is actually one of the few places where Javashows its limits performance-wise. So do other languages, as evidencedby the fact that FFmpeg developers actually use Assembler in a numberof places in order to handle video in the most efficient way possible.

10.7.4. Others

There are a number of other places where we've decided that we neededto go native for better results. Systray notifications with Growl onMac OS X and libnotify on Linux are one such example. Others includequerying contact databases from Microsoft Outlook and Apple AddressBook, determining source IP address depending on a destination, usingexisting codec implementations for Speex and G.722, capturing desktopscreenshots, and translating chars into key codes.

The important thing is that whenever we needed to choose a nativesolution, we could, and we did. This brings us to our point: Eversince we've started Jitsi we've fixed, added, or even entirelyrewritten various parts of it because we wanted them to look, feel orperform better. However, we've never ever regretted any of the thingswe didn't get right the first time. When in doubt, we simply pickedone of the available options and went with it. We could have waiteduntil we knew better what we were doing, but if we had, therewould be no Jitsi today.

用的OSS。

我们决定使用另一种音频系统。我们不想在多平台的性质上妥协，如果可能的话，我们希望避免自己独自处理。这是portaudio2非常方便的地方。

当Java不让您自己做一些事情时，跨平台的开源项目是下一个最好的选择。切换到PortAudio使我们能够实现对细粒度可配置审计和捕获的支持，正如我们在上面描述的那样。它还运行在windows、Linux、Mac OS X、FreeBSD以及其他一些我们没有时间为其提供包的东西上。

10.7.2。视频捕捉和呈现

视频对我们来说和音频一样重要。然而，对于Java的创建者来说，这似乎并不是这样的，因为在JRE中没有允许捕获或呈现视频的defaultAPI。有一段时间，Java媒体框架似乎注定要成为这样一个api，直到Sun停止维护它。

自然，我们开始寻找一个波特式的视频替代品，但这次我们没那么幸运了。一开始，我们决定使用KenLarson3的民用框架。这是一个非常棒的项目，我们用了很长一段时间。然而，在实时通信环境中使用时，它被证明是次优的。

因此，我们得出结论，为Jitsi提供完美的视频通信的唯一途径是让我们自己来实现本土的人和渲染器。这并不是一个简单的决定，因为它暗示了增加了大量的复杂性和对项目的维护负载，但是我们别无选择:我们真的想要有质量的视频通话。现在我们做的!

我们的本地graber和渲染器直接在Linux、Mac OS X和Windows上分别使用Video4Linux 2、qtkit以及direct3d/direct3d。

10.7.3。视频编码和解码

SIP通信者，因此Jitsi，在第一天就支持了视频通话。这是因为Java媒体框架允许使用h来编码视频。263编解码器和176x144(CIF)格式。那些知道h的人。到目前为止，CIF看起来可能是在微笑;如果这是它所能提供的一切，我们今天就会使用视频聊天应用。

为了提供像样的质量，我们不得不使用其他类似的库。视频编码实际上是少数几个在性能方面有限制的地方之一。其他语言也是如此，因为FFmpeg开发人员实际上在很多地方都使用汇编程序，以便以最有效的方式处理视频。

10.7.4。其他人

还有很多其他的地方，我们已经决定，为了更好的结果，我们需要本地化。在Linux上，带有咆哮的onMac OS X和libnotify的sy迷路通知就是这样一个例子。其他包括查询来自微软Outlook和苹果通讯录的联系人数据库，根据目的地确定源IP地址，使用现有的Speex和g的编解码器实现。捕获桌面截屏，并将chars转换为关键代码。

重要的是，无论何时我们需要选择一种本土文化，我们都能做到，而且我们做到了。这让我们想到了我们的观点:自从我们开始了Jitsi以来，我们已经固定了，添加了，甚至是重新编写了各种各样的部分，因为我们想让他们看起来，感觉或表现得更好。然而，我们从来没有后悔过我们第一次没有得到正确的东西。当有疑问时，我们只是简单地做了一些可用的选择，然后就去做了。我们可以一直等待，直到我们知道我们在做什么，但如果我们有的话，今天就不会有Jitsi了。

10.8. Acknowledgments

Many thanks to Yana Stamcheva for creating all the diagrams in thischapter.

Footnotes

1. To refer directly to thesource as you read, download it fromhttp://jitsi.org/source. If you are using Eclipse or NetBeans,you can go to http://jitsi.org/eclipse orhttp://jitsi.org/netbeans for instructions on how configurethem.
2. http://portaudio.com/
3. http://lti-civil.org/
4. Actually we still have it asa non-default option.