open binder

/*****************/
Binder Overview
The Binder defines and implements a distributed component architecture similar in broad strokes to COM on Windows and CORBA on Unix. That is, the system is defined as a set of abstract interfaces, operations on those interfaces, and implementations of them. As long as you are using these interfaces, the actual object behind them can be written in any language with Binder bindings, and exist anywhere your own address space, in another process, or even (eventually) on another machine.


Where did the name come from?
The Binder derives its name from its common use as a backbone to which you can attach a variety of separate pieces, creating a cohesive whole where the individual pieces all fit together.
This ability generally centers around the IBinder interface, the lingua franca of the Binder Kit, but begins with the Support Kit (providing common threading and memory management facilities) and extends up to the Binder Data Model (a rich language for describing complex, active data structures).
The Binder is a distributed architecture, so you generally don't worry about processes or IPC between them. The Binder infrastructure takes care of making all objects look like they exist in the local process, and dealing with IPC and other issues as needed. It also takes care of resource management between processes, so you are guaranteed that, as long as you have a handle on an object in another process, that the object will continue to exist. (Barring catastrophic situations such as a process crashing, but allowing other processes to deal gracefully with such the event.)
Unlike most other component architectures, the Binder is system oriented rather than application oriented. That is, the Binder is designed to support a new kind of component-based system-level development, and the current focus of its distributed architecture is on processes and IPC between them rather than cross-network communication. Indeed, network communication is not currently implemented, though that is not because of limitations in the architecture: it simply has not been the focus of development so far.


The Binder provides component/object representations of various basic system services, such as processes (through IProcess) and shared memory (through IMemory), allowing you to use them in new and more powerful ways. Unlike many attempts at rethinking how a system is designed, the Binder does not replace or hide these kinds of traditional concepts in operating systems, but instead attempts to embrace and transcend them. This allows the full Binder programming model to sit on top of any traditional operating system, and indeed in addition to Linux it has in the past run on BeOS, Windows, and Palm OS Cobalt.


The Binder is mostly implemented in C++, and currently that is the language you will do most of your programming in. However the Binder Shell can serve as another Binder language (you can even write Binder components with it), and because it exposes those same Binder facilities in a scripting language it can be a good way to learn about basic Binder concepts. The Binder Shell Tutorial provides such an introduction to the shell and Binder.


There are also many rich facilities for doing multithreaded programming with the Binder, though the Binder itself does not impose a particular threading model (in COM parlance, it is purely free threaded), It allows you to implement whatever threading model is appropriate for your situation, though we do use various Threading Conventions in our code built on top of the Binder. In addition, calls from one Binder object to another are always executed by having the thread in the first object simply hop over to the second. This model is extended to cross-process operations, providing the foundation for the rich Binder Process Model.


The Binder has been used to implement a wide variety of commercial-quality system-level services. A primary example is the implemention of a distributed user interface framework, where every node in the view hierarchy (including windows, layout managers, and controls) is a Binder object, allowing them to be distributed across processes as needed. It has also been used to implement a rich media framework (where the media graph is composed of Binder objects), an application manager, a font cache and font engines (where font engines are components for easy replacement and allowing applications to provide their own font engine of data for temporary use by the system), power management services, a rich list view control that sits on top of the Binder Data Model as a data source, etc.


Why Binder?
The Binder architecture was developed to address specific issues that were encountered while developing BeIA at Be, Inc and then Palm OS Cobalt at PalmSource, Inc. Both of these platforms were designed to run on small handheld or dedicated devices, an environment that imposes some specific requirements on system software that are not an issue in a more traditional desktop environment.
Hardware Scalability
The mobile device world tends to have a much broader range of hardware capabilities, for various reasons. For example, size and battery life are extremely important issues, so a new device may use less powerful hardware than an older device in order to meet these goals. This places a burden on system software, where one would like to be able to run on anything from a 50MHz ARM 7 CPU (without memory protection) up to a 400MHz ARM 9 CPU and beyond.
The Binder helps to address this situation by allowing for the creation of system designs that have much more flexibility in how they use hardware. In particular, memory protection and process communication is a significant overhead in modern operating systems, so the Binder strongly supports system design that is not tied to process organization. Instead, the Binder can assign various parts of the system to processes at run time, depending on the particular speed/size/stability/security trade-offs that make sense.


System Customization
Mobile and dedicated devices are unlike desktop machines in that, instead of the "one size fits all" world of the desktop, their user experience and functionality can vary widely between different devices. Both hardware manufacturers and phone carriers want to deeply customize their behavior, partly to support their branding, but also because these devices need to provide an experience that is more specific to the problems they are trying to solve rather than having a general purpose user interface.
The Binder's component model, applied to system design, makes it much easier to support this kind of customizability in a manageable way. The Palm OS Cobalt system architecture takes advantage of this by including separable components for things like font engines, key input handlers, power management, window management, and even UI focus indication. This has allowed, for example, a hardware vendor to customize how input focus is managed on-screen without having to write fragile code that is deeply tied to the implementation of the system software.


Robust Applications
The display and usage pattern of dedicated and mobile devices is also very different than it is on desktops. Desktop-style window management is generally not practical or, if there is no touch screen, even impossible. Because of the limited amount of space on the screen, the currently active application generally wants to consume as much of the available space as possible, essentially becoming the user experience.
This usage of the screen can become a significant problem for more complicated applications, such as web browsers. Often a web browser will need to rely on various third party code to display rich content such as movies, code that is itself quite complicated and not under complete control of the browser. If that code happens to crash in a traditional system design, it will at the very least take with it the window it is in which on a mobile device usually means the entire user experience goes away.


The Binder helps address this issue by making it easy for one application to sandbox other parts of itself. For example, a web browser using the Binder could decided to create components for displaying movies and other complicated content in another process, so that if those components crash they will not disrupt the containing browser experience. At the same time, the Binder can easily revert back to using a single process (or only a few) depending on the capabilities of the hardware it is running on.


Binder Compared To...
Though the Binder is similar in broad strokes to most component-based systems, the one it is most like is probably Microsoft's COM. If you a familiar with COM, many of the concepts used in the Binder will also be very familiar, though the details can be quite different. Since COM is the most popular component system around, it makes sense to describe the Binder in terms of its differences from COM.
COM's baseline language is C, while the Binder's is C++. By leveraging the richer C++ language, which has many more facilities to represent the object-oriented concepts of a component system, the Binder APIs can be much easier to use than the equivalent APIs in COM. See Binder Recipes for examples, such as the extensive use of smart pointer template classes to completely eliminate manual reference counting.
COM's primary API is a static interface (IUnknown), while the Binder's core API is a dynamically typed protocol provided by IBinder. In COM, scripting support is provided through a special interface (IDispatch) required to support scripting, while all Binder objects are intrinsically scriptable. The Binder Shell Tutorial contains many examples of how this can be used. The static Binder interfaces used in C++ sit on top of that core scripting infrastructure, and are in fact completely optional. Not only does this provide a powerful environment for dynamically typed languages, but it can also be used in C++ for situations where it is more convenient to use dynamic or ad-hoc bindings. See, for example, BObserver.
SValue, the Binder's equivalent to COM's "variant" dynamic type, includes a number of additional features. It is not constrained to a fixed set of types, but simply operates on generic typed data blobs. In addition, it can easily hold complex data structures (of sets of mappings of any SValue type, including other sets and mappings), with a fairly simple C++ API for dealing with these structures.
In addition to events that can be generated from objects, the Binder also provides a simple dataflow-like mechanism for properties. This allows you to connect the property of one object to the property of another object, such that a change in the first value will cause a corresponding change to the second. This is useful, for example, to display dynamic data to the user such as a battery level.
The Binder was designed from its beginning to be multithreaded. It doesn't support (or impose) threading models like COM, but instead all objects are free threaded. It includes many rich tools for threaded programming, including the lightweight SLocker class, a powerful predictive deadlock detection tool (LOCK_DEBUG), and the SHandler class for scheduling timed messages out of the Binder's built-in thread pool.


/*****************/
Binder Terminology
Alphabetical
Binder
Binder Object
Binder Namespace
Component
Context
Inspect
Interface
Link
Package
Package Manager
Service
Conceptual
Binder
We use this word in two ways: The Binder refers to the overall Binder architecture; a Binder is a particular implementation of a Binder interface. The name "Binder" comes from its ability to provide bindings to functions and data from one language or execution environment to another.
Binder Object
Any entity that implements IBinder, the core Binder interface. In practice, this generally means a class that derives from BBinder, the standard base implementation of IBinder. A Binder object is able to execute the core Binder operations: inspect, effect (a generalization of put/get/invoke), link, etc.
"Binder Object" is a very general concept: it says nothing about the language used, how the object was created (through the Package Manager or just manually by calling C++ new), the process it lives in, etc. It is simply something that has implemented IBinder, however that may be.


What is the difference between "Binder object" and "a Binder"? Any class that implements IBinder is a Binder object; that class may implement IBinder multiple times for each of the interfaces it supports, and each of those is "a Binder". So a Binder object may have muliple Binders. For example, see BIndexedDataNode, a class that contains two Binders: one for its INode interface, and one for its IIterable interface.


Component
A Binder Object that has been published in the Package Manager. Components are instantiated with SContext::New() or SContext::RemoteNew(), allowing both late binding to the implementation and the instantiation of components in other languages and processes. This is in contrast to instantiating a Binder Object with the C++ operator new, where you must link to a specific C++ implementation and instantiate the object in your local process.
Components use a Java-style naming scheme, such as com.palmsource.apps.AddressBook. Most system components are in the org.openbinder.* namespace.


Package
A Package contains one or more Component implementations. The key elements of a package are (1) the executable code implementing those components, and (2) a "manifest file" that describes the components implemented by that code. The code of a package exports a function called InstantiateComponent(), which is the factory for the components it implements. The Package Manager scans through all of the manifest files in the system to collect information about all of the available components, and returns the information associated with a component as needed.
Note that unlike COM, component information is maintained statically. There is not a Registry-like entity that keeps track of information about the components; instead, a component's information is constructed as needed directly from the manifest file that is part of a package.


Package Manager
The Package Manager is a sub-system of the Binder responsible for keeping track of the available components and implementing the dynamic component instantiation and management facilities.
There are actually two distinct parts to the Package Manager:


The Package Manager service, published in /packages in the Binder Namespace, is responsible for collecting package information and providing access to it through the Binder Namespace. In particular, it presents under /packages/components all of the available components and information associated with them that is needed to load and instantiate them in a process.
The SPackage, SPackageSptr, SSharedObject, and BProcess classes are responsible for loading package code into a process, managing its lifetime, providing access to resources in a package, and instantiating objects from it.
Finally, the SContext::New() API brings these pieces together two provide the Binder's formal component instantiation facility: it queries the Package Manager service for information about the requested component, and then calls IProcess::InstantiateComponent() to have that component loaded and instantied in the desired process.


Interface
A well-defined set of methods, properties, and events that a Binder can implement. These are usually described in IDL (see interfaces/...), converted by the pidgen tool to a C++ header and implementation. In C++, instead of deriving directly from BBinder, you will usually derive from some Bn* class generated by pidgen (such as BnDatum), which gives you a BBinder that is configured to implement a particular interface.
Inspect
Every Interface has an associated Binder. A Binder Object that implements multiple interfaces will thus have multiple Binders, one for each of its interfaces. You cast between these Binders/Interfaces using the IBinder::Inspect() call. Binder Inspect() Details has more details on this process.
Binder Namespace
The Binder Namespace is a hierarchical orginization of Binder objects. It is constructed using the Binder Data Model interfaces. Directories implement INode, and may implement any other interfaces for additional capabilities (almost always IIterable, very often ICatalog). Leaf objects implement IDatum if they contain a flat data stream (such as a file), but are not required to do so and often implement other interfaces.
Context
Each Binder Object is created in a "context", represented by the SContext object. The context is the thing that holds all of the global state the object can access services, settings, information about components it can instantiate, etc. The SContext you are running in is easily accessible through the BBinder::Context() method. For example, to find the Window Manager service, you would write code within your Binder Object along the lines of Context().LookupService(SString("window")).
SContext is, in fact, just holding the root INode of a Binder Namespace, providing convenience functions for doing common operations with the namespace. Components usually don't get an SContext of the root namespace to enforce security, the system makes additional namespaces that are derived from the root namespace but with modifications such as "can not publish new services".


Service
A previously instantiated Binder object that has been published under /services in the Binder Namespace. These are usually instantiated by the boot script when the system starts up, and exist forever.
You can basically think of them as singleton components, though the Binder does not currently provide any explicit way to define singletons. That is, there is no way to say "this component is a singleton" so that every attempt to instantiate the component returns the same object; instead, you must explicitly instantiate it and publish it in the namespace up front, for others to discover there.


Link
A Link is an active data/event connection from one Binder to another. It is created with the IBinder::Link() method, expressing a binding going from the Binder being called to the given target. The source Binder uses BBinder::Push() to send data through a link.
There are two kinds of links, data and events. All Interface properties are also data links, providing a data-flow mechanism. This can be used, for example, to attach the current value of some property in one object to the value of some other propety in a differnt object; when the value of the former changes, that new value will be propagated to the later.


Event links are created explicitly in IDL in the "event:" section. Events are linked to methods on a target, resulting in a method invocation when the event is pushed.


/****************/
Programming Guide
The Binder architecture is divided into a number of separate subsystems, generally building on top of each other. Note that this documentation is divided into a set of "kits" that is cleaner than how the current source tree is organized.
Support Kit: a set of standard tools and APIs used to write Binder code. This includes basic types such as strings, container classes such as vectors, tools for reference counting and memory management, and threading utilities.
SAtom Debugging
SHandler Patterns
Binder Kit: the core Binder APIs, defining Binder objects, interfaces, and other concepts.
Binder Recipes
Binder Shell Tutorial
Binder Shell Data
Binder Shell Syntax
Binder Terminology
Binder Process Model
Binder IPC Mechanism
pidgen
Binder Inspect() Details
Storage Kit: APIs for accessing and manipulating data.
Binder Data Model
Writing Data Objects


/*************/
Binder Recipes
This page provides simple recipes for common things you will do with the Binder. These serve as a good first look at what it is like to use the Binder's C++ APIs.


Calling a Binder Service
Writing a Binder Object
Giving a Binder Object to a Service
Creating a Component
Calling a Binder Service
The first thing you will usually want to do is call on to an existing Binder Service. In this example, we will use the Informant service to broadcast a message to others who may be interested.
Using a Binder Interface
#include
Binder interfaces are described through IDL, and the pidgen tool used used to convert them to a C++ API. To use an interface, you will need to include the C++ header that pidgen generated.


The Informant interface is located at "interfaces/services/IInformant.idl", so the header you need to include is "services/IInformant.h".


Contact the Service
sptr informantBinder
    = Context().LookupService(SString("informant"));
Every Binder object is associated with an SContext, which is your global connection with the rest of the system. All Binder objects have BBinder as a base class, so from the code inside your object you can retrieve your context through the BBinder::Context() API.


The SContext class includes various methods for making use of the information in the context. Here, we use SContext::LookupService() to retrieve, by name, a reference on a service that was previously published. The object we get back is a pointer to an IBinder, which is the abstract interface to any Binder Object.


Note the use of sptr instead of a raw C++ pointer. You should always use sptr<> (or wptr<>) when working with Binder objects, because they are reference counted and these classes will automatically hold a reference on the object for you.


Get a Binder Interface
sptr informant
    = interface_cast(informantBinder);
The interface_cast<> method finds an IDL Interface associated with a Binder object. In this case we are looking for the IInformant interface. This is conceptually the same as QueryInterface() in COM. In the Binder, it asks the object whether it supports that interface and for the IBinder associated with it, and returns the C++ interface requested. If the object is not in the local process, a proxy is created for it and returned.


The result of interface_cast<> is a sptr<> to the requested C++ interface or NULL if the object does ot implement that interface.


Call the Binder Object
if (informant != NULL) {
    informant->Inform(  SValue::String("myMessage"),
                        SValue::String("myData") );
}
We now have a fully functional C++ interface. It is pointing to the actual target object if this is a local C++ class, or a proxy object to the implementation if it is somewhere else.


In either case, we can make calls on the interface just like on a local C++ class. The object will stay around as long as we hold a sptr<> referencing the interface.


In this case, we will call the IInformant::Inform() method to broadcast a notification. The two arguments supply the information to be broadcast, which is a Binder type called an SValue. This is like a variant in COM or GValue in GLib: it carries a type as well as data, for dynamic typing. Here we are supplying two SValues holding string data, giving the message name and data to send.


Alternative to Contact the Service
Sometimes you are not a Binder object, but have some other code that needs to make use of the Binder. In this case, you need a way to get an SContext object through which you can find other Binder objects. This can be done through the SContext::UserContext() API.
SContext context = SContext::UserContext();
    
sptr informantBinder =
    context.LookupService(SString("informant"));
Note that SContext::UserContext() should only be used when you don't otherwise have an SContext available through BBinder or some other facility. You will normally want to use the SContext that was provided to you, since your caller may have customized it in some way.


Complete Example: Calling a Binder Service
To review, here is our complete example of calling a Binder service:
#include


sptr informantBinder
    = Context().LookupService(SString("informant"));


sptr informant
    = interface_cast(informantBinder);


if (informant != NULL) {
    informant->Inform(  SValue::String("myMessage"),
                        SValue::String("myData") );
}
Writing a Binder Object
We are now going to look at what you do to write your own Binder object that others can call.
Implementing a Binder Interface
#include
The interface we are going to implement is IInformed, a part of the Informant service, and so defined in the same file "services/IInformant.h" we saw above. The IInformed interface is the recipient of a message broadcast through IInformant.


Note that another common way to broadcast information is through a Binder Link, which allows you to declare properties and events on an interface that can be monitored by other objects. See Links for an example.
Define the Object Class
class MyWatcher : public BnInformed, public SPackageSptr
{
public:
MyWatcher is the class we are implementing.


The BnInformed class is generated by pidgen from the IInformed interface in the IInformant.idl file. BnInformed is a subclass of IInformed, which provides the basic mechanism for implementing a concrete IInformed class. In particular, it includes a BBinder object: this gives you an IBinder API that can be used by other languages and processes, and BnInformed implements the unmarshalling code to let those clients call your IInformed implementation.


The SPackageSptr class is a special part of the Package Manager. It is essentially a "strong pointer" to your code package, allowing it to monitor how your package is being used to ensure that your code stays loaded as long as it is needed. This also gives your implementation access to your associated SPackage object, through which you can retrieve resources such as strings and bitmaps.


Implement Constructor
    MyWatcher(const SContext& context)
        : BnInformed(context)
    {
    }
This is a typical implementation of a Binder object's constructor. It takes the SContext that the object is being instantiated in, and hands that off to its base classes. Doing this allows it to later use BBinder::Context() to retrieve that context.


Implement the IInformed API
    void OnInform(  const SValue& information,
                    const SValue& cookie, const SValue& key)
    {
IInformed has a single method, IInformed::OnInform(). All methods and properties on an interface base class are pure virtuals, so you must implement them to have a concrete class that can be instantiated.


We see SValue, our typed data container, here again as the arguments this method receives. In addition to simple typed data, an SValue can contain complex data structures and mappings of other SValues. This is a convenient facility to propagate whatever data we would like through the informant.


Implement OnInform() Method
        bout << "Got informed: " << information << endl;
    }
};
Your implementation of an interface method can do whatever it wants, just like any other C++ class. In the implementation here, we are going to use a Binder formatted text output stream to print the arguments we received.


The "bout" object is an ITextOutput stream that writes to the standard output stream. Most Binder objects (SValue here, and also SString, SMessage, etc) can be written as text to a text output stream like "bout". It is basically like the standard C++ output streams, but has some additional features for indentation, tagging lines, and managing the output of multiple concurrent threads. (See TEXT_OUTPUT_FORMAT for runtime options to control text stream output.)


Complete Example: Writing a Binder Object
To review, here is our complete example of writing a Binder object
#include


class MyWatcher : public BnInformed, public SPackageSptr
{
public:


    MyWatcher(const SContext& context)
        : BnInformed(context)
    {
    }


    void OnInform(  const SValue& information,
                    const SValue& cookie, const SValue& key)
    {
        bout << "Got informed: " << information << endl;
    }
};
Giving a Binder Object to a Service
A key aspect of the Binder is transfering object references between interfaces. When those interfaces are remote, this is how you set up communication paths between languages and processes even though, to the client, all of the work necessary to do this is invisible.
Get the Informant Service
sptr  informant =
    interface_cast(
        Context().LookupService(SString("informant")));
Just like we saw in the first example, the first thing we need to do is retrieve the informant service. Note that error checking is only needed at the end interface_cast<> will gracefully propagate errors. Also, none of the Binder APIs currently use exceptions to propagate error conditions.


Instantiate New Binder Object
sptr informed = new MyWatcher;
Here we create a new instance of the MyWatcher class that was previously implemented.


Note again the sptr<> smart pointer class. It will hold a reference on the object, and reference counting is designed so that we can directly assign a newly created object to a sptr<> without ever having to worry about reference counts.


The pointer we have her could just as well have been a sptr, but all we need later on is the IBinder class so that is what we decided to use.


Give Object to Informant
if (informant != NULL && informed != NULL)
{
    informant->RegisterForCallback(
        SValue::String("myMessage"), informed,
        SValue::String("OnInform"));
}
After doing the appropriate error checking, we will call the informant's IInformant::RegisterForCallback() API to give our object to it. At this point the informant now has a reference on MyWatcher, and if they are in different processes a remote connection has been established. The MyWatcher instance we created will remain around as long as the informant holds a reference on it.


The SValue::String("OnInform") parameter we pass in specifies the method for the informant to call. The informant uses the Binder's scripting protocol (which is a basic capability of any Binder object) to call any method on the IBinder that was given to it. When it does this, your OnInform() method will execute just as it would if a C++ program had called it directly.


Complete Example: Giving a Binder Object to a Service
To review, here is our complete example of giving a Binder object to a service.
#include


sptr  informant =
    interface_cast(
        Context().LookupService(SString("informant")));


sptr informed = new MyWatcher;


if (informant != NULL && informed != NULL)
{
    informant->RegisterForCallback(
        SValue::String("myMessage"), informed,
        SValue::String("OnInform"));
}
Creating a Component
Let's now turn the Binder object we wrote (MyWatcher) into a Binder Component.
A component is a Binder object that is published so that others can instantiate it, without having to link to its implementation. The Package Manager keeps track of all Binder components and the information needed to instantiate them.


In addition to the MyWatcher implementation, there are two more things you need to add to turn it into a full Binder component:


A Manifest.xml file, an XML document that tells the package manager about your component.
An InstantiateComponent() function, the factory function that generates instances of your component.
In addition, we will need a Makefile that puts all of this stuff together into the appropriate package structure.


MyWatcher's Manifest
The manifest for our component will be very simple:





This manifest says that there is a single component in the package, which implements the interface org.openbinder.services.IInformed. The name used to instantiate the component will simply be the package name itself, which we will define later in our Makefile.


The Factory Function
#include


sptr
InstantiateComponent(   const SString& component,
                        const SContext& context,
                        const SValue &args)
{
This is the function exported from your package, which the Package Manager calls when it needs to make a new instance of one of your components. Note that we include the "support/InstantiateComponent.h" header, which declares the InstantiateComponent() function so it will be properly exported.


The component argument you receive is the local name of the desired component, that is the full component name with the package name at the beginning removed.


The context argument is the Binder context the component is being instantiated in.


The args are data to be supplied to the component constructor. That is, it will contain an SValue of key/value mapping pairs supplying the arguments.


    if (component == "")
        return static_cast(new MyWatcher(context));
    return NULL;
}
Your implementation of InstantiateComponent() will need to look at the component argument to determine which component to instantiate. Here, we have only one component we have implemented. The component's name is the same as the package name, so the component argument here for that component will be "" that is, there is no suffix beyond the base package name for the component.


The standard C++ new operator is used to make a new instance of the component. Note that we pass the context to the MyWatcher class's constructor so it can later retrieve it through BBinder::Context() if needed.


The cast here is used to select the default interface that is returned. In this case it is not actually needed, because we are only implementing a single interface.


Note sptr<> again here, used implicitly due to the function return type, taking care of all reference counts for us.


If the requested component is not one we implement (which shouldn't happen unless our manifest file is incorrect), we simply return NULL.


MyWatcher's Makefile
LOCAL_PATH:= $(call my-dir)


include $(CLEAR_VARS)


BASE_PATH:= $(LOCAL_PATH)
PACKAGE_NAMESPACE:= org.openbinder.samples
PACKAGE_LEAF:= MyWatcher
SRC_FILES:= \
MyWatcher.cpp


include $(BUILD_PACKAGE)
This is an example of a Makefile that will use the OpenBinder build system to build the MyWatcher component into a Binder package. Note in particular the PACKAGE_NAMESPACE and PACKAGE_LEAF variables, which together set the base page name in this case, org.openbinder.samples.MyWatcher.


This Makefile will result in the creation of a directory build/packages/org.openbinder.samples.MyWatcher that contains:


MyWatcher.so the code implementing the MyWatcher package.
Manifest.xml the Manifest.xml file described previously.
The Package Manager uses the directory name to infer the base name for your package.


Instantiating a MyWatcher Component
Now that we have a component, we can use the SContext::New() API to create an instance of it.
sptr  informedBinder =
    Context().New(SString("org.openbinder.samples.MyWatcher"));
    
sptr informed =
    interface_cast(informedBinder);
The result of SContext::New() is an IBinder, the Binder object of the newly instantiated component, which you can then cast to the desired interface.


The above code can be done from any process. The component implementation will be loaded into the local process, if needed. In addition, the component can be implemented in another language or the system can decide it should be instantiated in a different process; in either case, proxies for IPC and/or marshalling will be created as needed, and you have no need to be aware that this has happened.


Objects vs. Components
Compare the previous code for directly instantiating a MyWatcher object:
sptr informed = new MyWatcher;
Using SContext::New() gives you the same thing: it works and behaves just like a local object. However, you now don't have to know about the component implementation up-front (linking directly to a specific library implementing it), and SContext::New() works across processes and languages.


Note, however, that the following code is not possible with SContext::New():


sptr informed = new MyWatcher;
In particular, though in some specific cases, the following may work (when the instantiated component is in the same process and language), this should not be done:


sptr informed = dynamic_cast(
    Context().New(SString("org.openbinder.samples.MyWatcher")).ptr();


/********************/
Binder Process Model
One could argue that you can't really understand what the Binder is about until you understand how it deals with processes. Conceptually, the Binder deals with processes in a very different way than you are probably used to thinking about them. This approach to processes is, in a way, its purest expression of the system-level component model it provides.


The process model described here is made possible by the underlying Binder IPC Mechanism features that allow operations remote objects to behave like local functional calls.


This IPC model is important to the Binder because it provides a high degree of process transparency. That is, developers for the most part don't have to worry about which process a particular Binder Object exists in, but can simply treat any instance as if it is in the local process. This allows for a lot of flexibility and scalability of systems built on top of the Binder, in that we can decided dynamically at run-time how they are to be distributed across processes (if any).


One immediately obvious benefit of this flexibility is that the entire Binder system is able to run without the process model being available at all. If the kernel driver implementing the Binder IPC Mechanism is not available, it will simply run everything in a single process. Likewise, the BINDER_SINGLE_PROCESS environment variable can be set to have everything run in one process, for ease of debugging.


This flexibility in process distribution leads, as described here, to some fundamental advances in the ways that high-level programmers can think about processes.


Process Basics
Binder Process Issues
Creating and Using Processes
Process Lifetime
Dynamic Process Binding
Process Basics
The best way to describe how the Binder manages processes is in contrast to what a system typically does, so first we will review the traditional approach seen elsewhere.
Processes are used in an operating system to protect one piece of code from disruption due to misbehavior of another piece of code. What this typically means is that each running application is given its own process, so a problem in one application won't cause trouble for other running applications.


Thus, the act of starting an application means to the operating system, "create a new process for the application, load the application code into that process, and start a thread running in that process at the beginning of the application."


This model makes sense because applications are generally fairly large, independent entities: when one application is running, it very rarely needs to know or wants to care at all about what other applications are doing at the same time. The process it is running in gives it its own isolated world, through which it can be protected from others and others can be protected from it.


Binder Process Issues
There are many things working together to make the traditional process model described above actually happen:
The kernel is responsible for creating and managing the basic process mechanism.
The runtime is responsible for loading and managing code in a process.
The shell (graphical or command line) is responsible for making the user action "run an application/command" map to lower-level operation of creating a process and running the associated code there.
Note that the Binder process model is designed to sit on top of standard kernel and runtime process support, such as fork()+exec() on Unix. It does not supplant or modify those in any way, but simply uses them in a different way than is done by traditional shells.


Unix actually provides a superset of the traditional process model being discussed. Calling fork() without an exec() will cause a new process to be created that is a complete duplicate of the current process. The Binder does not preclude using this alternative model alongside it, however the semantics of forking a multithreaded program are problematic enough that it is not a process model the Binder needs to support for itself. For example, on Linux a fork() does not duplicate any other threads in the original process, leaving parts the new process in a fairly questionable state.
In the Binder itself, there is no concept of an "application" an application is built on top of the core Binder services, but is not something the Binder itself defines. Instead, the Binder revolves around smaller, more generic components. An application is one specific kind of component you can create with the Binder, but you can just as well create components that represent controls on the screen, file-like data streams, a window manager, etc. To the Binder itself, these are all the same thing.
The goal, then, is to define a process model for the Binder that is independent of an application. Clearly, we can't just treat every single component someone creates as a potential application and give it its own process, so some other approach must be taken.


Creating and Using Processes
Processes in the Binder are modelled as sandboxes or containers in which to run components. Instead of having some code you want to run and then creating a process in which to run it, the Binder's approach is to first create an empty process, into which you can then instantiate any arbitrary code components.
In fact, a process in the Binder can best be thought of as just another kind of component. There is a special API you call to create it, but what you get back is just another Binder interface (called, not surprisingly, IProcess) that then allows you to interact with your new process.


Here is some example Binder code that will make a new process, and then create a new component inside of that process:


sptr process = Context().NewProcess(
    SString("my empty process"));


sptr obj = Context().RemoteNew(
    SValue::String("org.openbinder.services.WindowManager"), process);
Or the same thing from the Binder Shell:


/$ p=$[new_process "my empty process"]
Result: IBinder::shnd(0x2)


/$ c=$[new -r $p org.openbinder.services.WindowManager]
Result: IBinder::shnd(0x3)
After running this code, you now have a Window Manager service running in its own process. Note that if the service being instantiated is an application, the end result would be the same as with a traditional process model. For example:


sptr process = Context().NewProcess(
    SString("AddressBook"));


sptr obj = Context().RemoteNew(
    SValue::String("com.palmsource.apps.AddressBook"), process);
Or:


/$ p=$[new_process "AddressBook"]
Result: IBinder::shnd(0x2)


/$ app=$[new -r $p com.palmsource.apps.AddressBook]
Result: IBinder::shnd(0x3)
With this code we have started a process running the Address Book application/component.


With the Binder, however, we have a lot more flexibility in how we use processes. For example, we can run a number of related components in the same process in order to reduce the overhead needed for them:


sptr process = Context().NewProcess(
    SString("my empty process"));


sptr surface = Context().RemoteNew(
    SValue::String("org.openbinder.services.Surface"), process);


sptr wm = Context().RemoteNew(
    SValue::String("org.openbinder.services.WindowManager"), process);
Or:


/$ p=$[new_process "my empty process"]
Result: IBinder::shnd(0x2)


/$ surface=$[new -r $p org.openbinder.services.Surface]
Result: IBinder::shnd(0x3)


/$ wm=$[new -r $p org.openbinder.services.WindowManager]
Result: IBinder::shnd(0x4)
The division between processes can be done completely at run-time, depending on factors such as the capabilities of the hardware, the permissions needed by components, and third party components installed on the device.


Process Lifetime
In addition to deciding what process to spawn and how to distribute components between them, we also need to decide when those processes go away.
The default rule for the Binder is that a process lasts as long as there are any strong references on its objects that are held by other processes. In the most trivial case, this means that after creating an empty process you can make it go away by releasing your reference on that object:


/$ p=$[new_process "my temporary process"]
Result: IBinder::shnd(0x4)


/$ p=
Result: ""
[SIGCHLD handler] child process 23356 exited normally with exit value 0
More interesting, if you create any components in that process, then it will continue to stay around as long as other processes are using those components:


/$ p=$[new_process "my temporary process"]
Result: IBinder::shnd(0x2)


/$ c=$[new -r $p org.openbinder.tools.commands.BPerf]
Result: IBinder::shnd(0x4)


/$ p=
Result: ""


/$ c=
Result: ""
[SIGCHLD handler] child process 23359 exited normally with exit value 0
This is a very powerful feature, as it allows you to easily create temporary processes as sandboxes, which will automatically stay around for only as long as they are needed. An example of how you could use this feature would be in a web browser: if you have some code that you don't want running in the main browser process (such as a third party extension for displaying a special kind of media), you can create a temporary process in which it will run, and as long as others are using that code the process will continue to exist.


Even better, your web browser can hold a weak reference on that temporary process. Then, whenever it has new content to display, it can try to promote that reference and if the promotion succeeds then it can continue using the process for creating additional instances of the component. Otherwise, the current process has exited and it will need to make a new process.


Sometimes, however, you want to have more control over the lifetime of a process. This can be accomplished with the SLooper::StopProcess() method or stop_process shell command. These ask that the process stop as soon as all strong references on its process object go away, allowing you to let the process go away even if others have references to some of its components. In addition, you can specify an option to force the process to exit immediately, even ignoring anyone holding references on the process object.


See Process Lifetime in the Binder Shell Tutorial for more examples of this feature.


Dynamic Process Binding
The normal way you instantiate a component is with the SContext::New() method:
sptr obj = Context().New(
    SValue::String("com.vender.MyApp.Something"));
Unlike the RemoteNew() call, New() does not specify the process the new component should be created in. What this usually means is just that the component will always be instantiated in the local process. However, because the Binder hides the details of which processes components live in, it is not required to use the same process.


A simple form of this dynamic process binding can be seen with the BINDER_SINGLE_PROCESS option. This is a fairly brute-force approach, which simply forces SContext::NewProcess() to not create a new process unless the caller specifies that one is absolutely necessary. In other words, "run as much of the system as possible in a single process."


The current OpenBinder distribution also includes an early implementation of more extensive process management. This is provided through the IProcessManager interface, and a simple implementation of that API which currently lives within smooved itself.


This implementation allows a component to say that it would like a dedicated process for its package, and have itself instantiated there instead of the local process of the caller. The example code in samples/ServiceProcess demonstrates this functionality, as can be seen in the Dynamic Processes section of the Binder Shell Tutorial.


As mentioned, though, the current implementation is quite primitive and with the underlying architecture there is are many more interesting things that could be done. For example, the Process Manager could look at the signature of the component's executable and permissions needed by the component itself and, if they are in conflict with the current process, behind the scenes create the component in a more appropriate process. By doing so, we can enable a number of useful scenarios for dealing with security and other protection issues:


The current process is very trusted (complete access to the entire system), but calls New() to create a component that is not correctly signed to be run in such an environment. The Process Manager creates a new untrusted process and instantiates the component there instead.
The current process is completely untrusted, but calls New() to create a component that needs access to the telephone module. The Process Manager checks the signature of the new component's code and, seeing that it is safe, instatiates the new component in an existing process that is used for telephony-related functions.
The current process is completely untrusted, but calls New() to create a component that needs access to the telephone module. That component also specifies that it can only be used by others who also are allowed access to the telephone module. The Process Manager sees that the current process is not allowed this functionality and fails the instanstiation.
The current process has the same permissions as a component being instantiated. However, that component's manifest says that it would like to be run in its own process. On a high-end device, the Process Manager creates a new process for it and returns a remove object. On a low-end device, it instantiates the component in the local process.
The current process has the same permissions as a component being instantiated. However, that component's manifest says that all of the components in that package should run in the same process. There is already another component from that package running in a different process, so the Process Manager finds that other process and instantiates the new component there. (Hey, we now do this one! Progress!)




/×××××××××××××××××/
Binder IPC mechanism
The Binder communicates between processes using a small custom kernel module. This is used instead of standard Linux IPC facilities so that we can efficiently model our IPC operations as "thread migration". That is, an IPC between processes looks as if the thread instigating the IPC has hopped over to the destination process to execute the code there, and then hopped back with the result.


The Binder IPC mechanism itself, however, is not actually implemented using thread migration. Instead, the Binder's user-space code maintains a pool of available threads in each process, which are used to process incoming IPCs and execute local events in that process. The kernel module emulates a thread migration model by propagating thread priorities across processes as IPCs are dispatched and ensuring that, if an IPC recurses back into an originating process, the IPC is handled by its originating thread.


In addition to IPC itself, the Binder's kernel module is also resposible for tracking object references across processes. This involves mapping from remote object references in one process to the real object in its host process, and making sure that objects are not destroyed as long as other processes hold references on them.


The rest of this document will describe in detail how Binder IPC works. These details are not exposed to application developers, so they can be safely ignored.


Getting Started
When a user-space thread wants to participate in Binder IPC (either to send an IPC to another process or to receiving an incoming IPC), the first thing it must do is open the driver supplied by the Binder kernel module. This associates a file descriptor with that thread, which the kernel module uses to identify the initiators and recipients of Binder IPCs.
It is through this file descriptor that all interaction with the IPC mechanism will happen, through a small set of ioctl() commands. The main commands are:


BINDER_WRITE_READ sends zero or more Binder operations, then blocks waiting to receive incoming operations and return with a result. (This is the same as doing a normal write() followed by a read() on the file descriptor, just a little more efficient.)
BINDER_SET_WAKEUP_TIME sets the time at which the next user-space event is scheduled to happen in the calling process.
BINDER_SET_IDLE_TIMEOUT sets the time threads will remain idle (waiting for a new incoming transaction) before they time out.
BINDER_SET_REPLY_TIMEOUT sets the time threads will block waiting for a reply until they time out.
BINDER_SET_MAX_THREADS sets the maximum number of threads that the driver is allowed to create for that process's thread pool.
The key command is BINDER_WRITE_READ, which is the basis for all IPC operations. Before going into detail about that, however, it should be mentioned that the driver expects the user code to maintain a pool of threads waiting for incoming transactions. You need to ensure that there is always a thread available (up to the maximum number of threads you would like) so that IPCs can be processed. The driver also takes care of waking up threads in the pool when it is time to process new asynchronous events (from SHandler) in the local process.


BINDER_WRITE_READ
As mentioned, the core functionality of the driver is encapsulated in the BINDER_WRITE_READ operation. The ioctl's data is this structure:
struct binder_write_read
{
    ssize_t     write_size;
    const void* write_buffer;
    ssize_t     read_size;
    void*       read_buffer;
};
Upon calling the driver, write_buffer contains a series of commands for it to perform, and upon return read_buffer is filled in with a series of responses for the thread to execute. In general the write buffer will consist of zero or more book-keeping commands (usually incrementing/decrementing object references) and ending with a command requiring a response (such as sending an IPC transaction or attempt to acquire a strong reference on a remote object). Likewise, the receive buffer will be filled with a series of book-keeping commands and end with either the result for the last written command, or a command to perform a new nested operation.


Here is a list of the commands that can be sent by a process to the driver, with comments describing the data that follows each command in the buffer:


enum BinderDriverCommandProtocol {
    bcNOOP = 0,
        No parameters! 


    bcTRANSACTION,
    bcREPLY,
    
        binder_transaction_data: the sent command.
    
    
    bcACQUIRE_RESULT,
    
        int32:  0 if the last brATTEMPT_ACQUIRE was not successful.
        Else you have acquired a primary reference on the object.
    
    
    bcFREE_BUFFER,
    
        void *: ptr to transaction data received on a read
    
    
    bcINCREFS,
    bcACQUIRE,
    bcRELEASE,
    bcDECREFS,
    
        int32:  descriptor
    
    
    bcATTEMPT_ACQUIRE,
    
        int32:  priority
        int32:  descriptor
    
    
    bcRESUME_THREAD,
    
        int32:  thread ID
    
    
    bcSET_THREAD_ENTRY,
    
        void *: thread entry function for new threads created to handle tasks
        void *: argument passed to those threads
    
    
    bcREGISTER_LOOPER,
    
        No parameters.
        Register a spawned looper thread with the device.  This must be
        called by the function that is supplied in bcSET_THREAD_ENTRY as
        part of its initialization with the binder.
    
    
    bcENTER_LOOPER,
    bcEXIT_LOOPER,
    
        No parameters.
        These two commands are sent as an application-level thread
        enters and exits the binder loop, respectively.  They are
        used so the binder can have an accurate count of the number
        of looping threads it has available.
    
    
    bcCATCH_ROOT_OBJECTS,
    
        No parameters.
        Call this to have your team start catching root objects
        published by other teams that are spawned outside of the binder.
        When this happens, you will receive a brTRANSACTION with the
        tfRootObject flag set.  (Note that this is distinct from receiving
        normal root objects, which are a brREPLY.)
    
    
    bcKILL_TEAM
    
        No parameters.
        Simulate death of a kernel team.  For debugging only.
    
};
The most interesting commands here are bcTRANSACTION and bcREPLY, which initiate an IPC transaction and return a reply for a transaction, respectively. The data structure following these commands is:


enum transaction_flags {
    tfInline = 0x01,            // not yet implemented
    tfRootObject = 0x04,        // contents are the component's root object
    tfStatusCode = 0x08         // contents are a 32-bit status code
};


struct binder_transaction_data
{
    // The first two are only used for bcTRANSACTION and brTRANSACTION,
    // identifying the target and contents of the transaction.
    union {
        size_t  handle;     // target descriptor of command transaction
        void    *ptr;       // target descriptor of return transaction
    } target;
    uint32  code;           // transaction command
    
    // General information about the transaction.
    uint32  flags;
    int32   priority;       // requested/current thread priority
    size_t  data_size;      // number of bytes of data
    size_t  offsets_size;   // number of bytes of object offsets
    
    // If this transaction is inline, the data immediately
    // follows here; otherwise, it ends with a pointer to
    // the data buffer.
    union {
        struct {
            const void  *buffer;    // transaction data
            const void  *offsets;   // binder object offsets
        } ptr;
        uint8   buf[8];
    } data;
};
Thus, to initiate an IPC transaction, you will essentially perform a BINDER_READ_WRITE ioctl with the write buffer containing bcTRANSACTION follewed by a binder_transaction_data. In this structure target is the handle of the object that should receive the transaction (we'll talk about handles later), code tells the object what to do when it receives the transaction, priority is the thread priority to run the IPC at, and there is a data buffer containing the transaction data, as well as an (optional) additional offsets buffer of meta-data.


Given the target handle, the driver determines which process that object lives in and dispatches this transaction to one of the waiting threads in its thread pool (spawning a new thread if needed). That thread is waiting in a BINDER_WRITE_READ ioctl() to the driver, and so returns with its read buffer filled in with the commands it needs to execute. These commands a very similar to the write commands, for the most part corresponding to write operations on the other side:


enum BinderDriverReturnProtocol {
    brERROR = -1,
    
        int32: error code
    
    
    brOK = 0,
    brTIMEOUT,
    brWAKEUP,
        No parameters! 
    
    brTRANSACTION,
    brREPLY,
    
        binder_transaction_data: the received command.
    
    
    brACQUIRE_RESULT,
    
        int32: 0 if the last bcATTEMPT_ACQUIRE was not successful.
        Else the remote object has acquired a primary reference.
    
    
    brDEAD_REPLY,
    
        The target of the last transaction (either a bcTRANSACTION or
        a bcATTEMPT_ACQUIRE) is no longer with us.  No parameters.
    
    
    brTRANSACTION_COMPLETE,
    
        No parameters... always refers to the last transaction requested
        (including replies).  Note that this will be sent even for asynchronous
        transactions.
    
    
    brINCREFS,
    brACQUIRE,
    brRELEASE,
    brDECREFS,
    
        void *: ptr to binder
    
    
    brATTEMPT_ACQUIRE,
    
        int32:  priority
        void *: ptr to binder
    
    
    brEVENT_OCCURRED,
    
        This is returned when the bcSET_NEXT_EVENT_TIME has elapsed.
        At this point the next event time is set to B_INFINITE_TIMEOUT,
        so you must send another bcSET_NEXT_EVENT_TIME command if you
        have another event pending.
    
    
    brFINISHED
};
Continuing our example, the receiving thread will come back with a brTRANSACTION command at the end of its buffer. This command uses the same binder_transaction_data structure that was used to send the data, basically containing the same information that was sent but now available in the local process space.


The recipient, in user space will then hand this transaction over to the target object for it to execute and return its result. Upon getting the result, a new write buffer is created containing the bcREPLY reply command with a binder_transaction_data structure containing the resulting data. This is returned with a BINDER_WRITE_READ ioctl() on the driver, sending the reply back to the original process and leaving the thread waiting for the next transaction to perform.


The original thread finally returns back from its own BINDER_WRITE_READ with a brREPLY command containing the reply data.


Note that the original thread may also receive brTRANSACTION commands while it is waiting for a reply. This represents a recursion across processes the receiving thread making a call on to an object back in the original process. It is the responsibility of the driver to keep track of all active transactions, so it can dispatch transactions to the correct thread when recursion happens.


Object Mapping and Referencing
One of the important responsibilities of the driver is to perform mapping of objects from one process to another. This is key to both the communication mechanism (targetting and referencing objects) as well as the capability model (only allowing a particular process to perform operations on remote objects that it has been explicitly given knowledge about).
There are two distinct forms of an object reference: as an address in a processes's memory space, or as an abstract 32-bit handle. These representations are mutually exclusive: all references in a process to objects local to that process are in the form of an address, while all references to objects in another process are always in the form of a handle.


For example, note the target field of binder_transaction_data. When sending a transaction, this contains a handle to the destination object (because you always send transactions to other processes). The recipient of the transaction, however, sees this as a point to the object in its local address space. The driver maintains mappings of pointers and handles between processes so that it can perform this translation.


We also must be able to send references to objects through transactions. This is done by placing the object reference (either a local pointer or remote handle) in to the transaction buffer. The driver must translate this reference to the corresponding reference in the receiving process, however, just like we do with the transaction target.


In order to do reference translation, the driver needs to know where these references appear in the transaction data. This is where the additional offsets buffer comes in. It contains of a series of indicies into the data buffer, describing where objects appear. The driver can then rewrite the buffer data, translating each of those objects from the sending process reference to the correct reference in the receiving process.


Note that the driver does not know anything about a particular Binder object until that object is sent through the driver to another process. At that point the driver adds the object's address to its mapping table and asks the owning process to hold a reference on it. When no other processes know about the object, it is removed from the mapping table and its owning process is told to release the driver's reference. This avoids maintaining the (relatively significant) driver state for an object as long as it is only used in its local process.


/************/
Support Kit: a set of standard tools and APIs used to write Binder code. This includes basic types such as strings, container classes such as vectors, tools for reference counting and memory management, and threading utilities.
SAtom Debugging
SHandler Patterns