Augmented Reality on Android: Using GPS and the Accelerometer

Augmented Reality is the ability to overlay location data points on the live view of a mobile device's camera. In a sense, AR allows the phone to become a window into a slightly different, data-driven world. 
To me, however, AR appears to be little more than a gimmick at this point. Before you fill my inbox with hate mail, let me quickly explain why:

1.The sensors involved (compass, accelerometer, and GPS) aren't nearly advanced enough to do the kind of real-time tracking required for a useable AR app.
2.No currently available data set performs best in this medium. That is to say, any data set you can show in AR today would look better in something like a Google Maps view.
However, any good programmer knows that the best time to enter a market is before a killer application makes its debut, rather than when the market is already proven. With that in mind, DevX will publish a two-article series to set you on the path to building your own Augmented Reality engine on Android. Building an AR application requires a dash of math and four technological pieces. This first article covers the first two pieces: the camera and the compass; the next article will cover the other two: the accelerometer and GPS.

The third step along the way to your Augmented Reality Engine on Android—after implementing the first two elements: the camera and the compass—is determining your location. You do this primarily through Android's LocationManager object. Before you can get one, however, you'll need to jump through a few hoops. Permissions, while annoying, are an important hurdle you'll need to clear. 
Android's LocationManager carries two permission requirements:

1.You need to tell the system you'd like to fetch the user's location.
2.You need to tell it that you want very detailed geographic information.
You request both permissions in the AndroidManifest.xml file with the <permission> tag inside the <manifest> tag as follows:

<uses-permission android:name="android.permission.LOCATION"/>For fine-grain location updates, which you'll need if the objects you want to display are nearby, you'll also have to add the following:

<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />Without these two lines in the manifest, when you attempt to register for location updates, Android will return a security exception that will stop your app dead. I don't believe I've written an Android application yet in which I haven't forgotten at least one permission request.

Getting the Location Manager
While getting the location manager may seem like a simple task at first, there are a few things you'll need to keep in mind. First, you can request the location manager only on the main UI thread. You'll either need to request your LocationManager object in the onCreate call of your activity or create a Runnable object with the LocationManager request to be run on the main thread. 
For the sake of simplicity, the following example code for registering for LocationManager updates runs from within an activity's onCreate method:

public void onCreate(Bundle savedInstanceState) 
{
   LocationManager locMan;
   locMan =
      (LocationManager)getSystemService(Context.LOCATION_SERVICE);
   locMan.requestLocationUpdates(LocationManager.GPS_PROVIDER, 
                           100, 1, gpsListener);   
}As you can see above, you declare your LocationManager object, use a call to getSystemService (a method on the Context class) with a cast to fetch your object, and use a method call to requestLocationUpdates.

You may wonder what parameters you pass into the location updates request. First, you tell the system you'd like location updates using the GPS unit in the system. Next, you tell it how often you'd like updates (in this case, every 100ms) and that you'd like an update if the person moves more than a meter. You ask for updates at this very high level of service because if the person moves you need to recognize their movement quickly and adjust their relationship to the other objects in space. Lastly, you' pass in an instance of a class that implements the LocationListener interface.

The Fine Art of Listening for Location Updates
After you pass off your request for location updates, the LocationListener class will be notified of any initial locations, followed by any subsequent location changes. Here's what your LocationListener should look like: 
LocationListener gpsListener = new LocationListener(){
      Location curLocation;
      boolean locationChanged = false; 
      public void onLocationChanged(Location location)
      {
         if(curLocation == null)
         {
            curLocation = location;
            locationChanged = true;
         }
         
         if(curLocation.getLatitude() == location.getLatitude() &&
               curLocation.getLongitude() == location.getLongitude())
            locationChanged = false;
         else
            locationChanged = true;
         
         curLocation = location;
      }
      public void onProviderDisabled(String provider){}
      public void onProviderEnabled(String provider){}
      public void onStatusChanged(String provider, int status, Bundle extras){}
};The only method in the above code that you're concerned about—at least for this proof of concept—is onLocationChanged. However, I included the rest of them in case you'd like to copy this object into your own code. The onLocationChanged method will be called as soon as the device locks up with the satellites and then on the intervals you specified in your request updates call (in this case, 100ms). Given the calculations involved in AR, I suggest having an optimization based on the locationChanged Boolean.

With each of the location updates comes a Location object. This class allows us to get the latitude and longitude of the subject, as well as do a host of important things. The methods of most interest in this discussion are getLatitude(), getLongitude(), bearingTo(), and distanceTo(). Using these four functions, you can optimize, figure out the azimuth (bearing in degrees east of north) to any subsequent location, and determine how far away you are (given another latitude and longitude point).

Requesting Accelerometer Data
The last piece you need to implement for your AR engine is access to the accelerometer data. Thankfully, Android has made this information easy to gather. In the previous AR article, the example called for requesting the orientation of the phone, and a call to registerListener on the location manager object retrieved the compass data. You use nearly the same technique to request accelerometer data, except of course you ask for accelerometer data with the following line: 
sensorMan = (SensorManager) ctx.getSystemService(Context.SENSOR_SERVICE);
sensorMan.registerListener(listener,
   sensorMan.getDefaultSensor(Sensor.TYPE_ACCELEROMETER),
   SensorManager.SENSOR_DELAY_FASTEST);

You call the getSystemService method on a context object (ctx in the above code). Here is the complete code for your orientation and the accelerometer listener.

private SensorEventListener listener = new SensorEventListener(){
   public static volatile float direction = (float) 0;
   public static volatile float inclination;
   public static volatile float rollingZ = (float)0;

   public static volatile float kFilteringFactor = (float)0.05;
   public static float aboveOrBelow = (float)0;

   public void onAccuracyChanged(Sensor arg0, int arg1){}

   public void onSensorChanged(SensorEvent evt)
   {
      float vals[] = evt.values;
      
      if(evt.sensor.getType() == Sensor.TYPE_ORIENTATION)
      {
         float rawDirection = vals[0];

         direction =(float) ((rawDirection * kFilteringFactor) + 
            (direction * (1.0 - kFilteringFactor)));

          inclination = 
            (float) ((vals[2] * kFilteringFactor) + 
            (inclination * (1.0 - kFilteringFactor)));

                
          if(aboveOrBelow > 0)
             inclination = inclination * -1;
          
         if(evt.sensor.getType() == Sensor.TYPE_ACCELEROMETER)
         {
            aboveOrBelow =
               (float) ((vals[2] * kFilteringFactor) + 
               (aboveOrBelow * (1.0 - kFilteringFactor)));
         }
      }
   }
};Yikes, that's a lot of code. What is going on here? First, you set up all the values for your listener. This means that at any time (like when a draw call passes by, for example) you can query the listener for the compass bearing and inclination of the phone. Each value comes in an array of integers. Those values will be different depending on the type of update you're getting (orientation or acceleration) You can see from the code which values to pull out when.

Next, you take the sensor data and use a little filtering math to determine two critical pieces of information:

1.The direction the phone is pointing
2.The screen's angle relative to the horizon
The first is called the Azimuth, or direction from north, and the second is called the inclination, or the angle above or below the horizon at which the camera is pointed. To determine these values, your first mathematical task is to filter the compass movement of the camera. This is called a rolling filter because of the length constraints. The direction variable tells you where the top of the phone is pointed, not where the camera itself is pointed, so you'll need to correct it a little.

Your second mathematical task is to run a rolling filter on the pitch, which will provide a measurement in degrees, where 90 is on the horizon, 45 is halfway up or halfway down, and 0 is straight up or straight down. Notice that with a reading of 45 you don't know whether the phone is pointed up or down from the horizon. This is where the accelerometer comes in. You use the reading generated by the accelerometer to make the inclination positive (above the horizon) or negative (below the horizon).

That, in a nutshell, is all you need from the acceleration sensor.

The Building Blocks of Augmented Reality
You now have all the tools required to build your own Augmented Reality engine. All that's left is a little math, some Android layout foo, and a fair amount of gumption. If you are more interested in building an AR app than the AR engine itself, I am currently putting together an open source Augmented Reality engine for Android. You can follow my somewhat faltering progress on Twitter at twitter.com/androidarkit. 
As much as I'd like to put all the math and drawing code together and unify these three pieces of information with an overlay on the camera, all that is beyond the scope of this article. However, this and the previous AR article should serve as complete introductions to the compass, camera preview, accelerometer, and GPS subsystems of Android. Now you have the building blocks you need to create the next big Augmented Reality application

 

----------------------------------------------------

Android defines a user space C abstraction interface for GPS hardware. The interface header is defined in include/hardware/gps.h. In order to integate GPS with Android, you need to build a shared library that implements this interface.

Building a GPS Library

To implement a GPS driver, create a shared library that implements the interface defined in gps.h. You must name your shared library libgps.so so that it will get loaded from /system/lib at runtime. Place GPS sources and Android.mk in vendor/acme/chipset_or_board/gps/ (where "acme" is your organization name and "chipset_or_board" is your hardware target).

The following stub Android.mk file ensures that libgps compiles and links to the appropriate libraries:

LOCAL_PATH := $(call my-dir) include $(CLEAR_VARS)

LOCAL_MODULE := libgps

LOCAL_STATIC_LIBRARIES:= \ # include any static library dependencies

LOCAL_SHARED_LIBRARIES := \ # include any shared library dependencies

LOCAL_SRC_FILES += \ # include your source files. eg. MyGpsLibrary.cpp

LOCAL_CFLAGS += \ # include any needed compile flags

LOCAL_C_INCLUDES:= \ # include any needed local header filesinclude $(BUILD_SHARED_LIBRARY)