Experiments in Streaming Content in Java ME(三)-----Back to RTPSourceStream and StreamingDataSource

With the protocol handler in place, let's revisit the RTPSourceStream and StreamingDataSource classes from earlier, where they contained only place-holder methods. The StreamingDataSource is simple to code:

import java.io.IOException;
import javax.microedition.media.Control;
import javax.microedition.media.protocol.DataSource;
import javax.microedition.media.protocol.SourceStream;

public class StreamingDataSource extends DataSource {

  // the full URL like locator to the destination
  private String locator;

  // the internal streams that connect to the source
  // in this case, there is only one
  private SourceStream[] streams;

  // is this connected to its source?
  private Boolean connected = false;

  public StreamingDataSource(String locator) {
      super(locator);
      setLocator(locator);
  }

  public void setLocator(String locator) { this.locator = locator; }

  public String getLocator() { return locator; }

  public void connect() throws IOException {

    // if already connected, return
    if (connected) return;

    // if locator is null, then can't actually connect
    if (locator == null)
      throw new IOException("locator is null");

    // now populate the sourcestream array
    streams = new RTPSourceStream[1];

    // with a new RTPSourceStream
    streams[0] = new RTPSourceStream(locator);

    // set flag
    connected = true;

    }

  public void disconnect() {

    // if there are any streams
    if (streams != null) {

      // close the individual stream
        try {
          ((RTPSourceStream)streams[0]).close();
        } catch(IOException ioex) {} // silent
    }

    // and set the flag
    connected = false;
  }

  public void start() throws IOException {

    if(!connected) return;

    // start the underlying stream
    ((RTPSourceStream)streams[0]).start();

  }

  public void stop() throws IOException {

    if(!connected) return;

    // stop the underlying stream
    ((RTPSourceStream)streams[0])Close();

  }

  public String getContentType() {
    // for the purposes of this article, it is only video/mpeg
    return "video/mpeg";
  }

  public Control[] getControls() { return new Control[0]; }

  public Control getControl(String controlType) { return null; }

  public SourceStream[] getStreams() {    return streams; }

}

The main work takes place in the connect() method. It creates a new RTPSourceStream with the requested address. Notice that the getContentType() method returns video/mpeg as the default content type, but change it to the supported content type for your system. Of course, this should not be hard-coded; it should be based on the actual support for different media types.

The next listing shows the complete RTPSourceStream class, which, along with RTSPProtocolHandler, does the bulk of work in connecting getting the RTP packets of the server:

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import javax.microedition.io.Datagram;
import javax.microedition.io.Connector;
import javax.microedition.media.Control;
import javax.microedition.io.SocketConnection;
import javax.microedition.io.DatagramConnection;
import javax.microedition.media.protocol.SourceStream;
import javax.microedition.media.protocol.ContentDescriptor;

public class RTPSourceStream implements SourceStream {

    private RTSPProtocolHandler handler;

    private InputStream is;
    private OutputStream Os;

    private DatagramConnection socket;

    public RTPSourceStream(String address) throws IOException {

        // create the protocol handler and set it up so that the
        // application is ready to read data

        // create a socketconnection to the remote host
        // (in this case I have set it up so that its localhost, you can
        // change it to wherever your server resides)
        SocketConnection sc =
          (SocketConnection)Connector.open("socket://localhost:554");

        // open the input and output streams
        is = sc.openInputStream();
        Os = sc.openOutputStream();

        // and initialize the handler
        handler = new RTSPProtocolHandler(address, is, Os);

        // send the basic signals to get it ready
        handler.doDescribe();
        handler.doSetup();
    }

    public void start() throws IOException {

      // open a local socket on port 8080 to read data to
      socket = (DatagramConnection)Connector.open("datagram://:8080");

      // and send the PLAY command
      handler.doPlay();
    }

    public void close() throws IOException {

        if(handler != null) handler.doTeardown();

        is.close();
        os.close();
    }

    public int read(byte[] buffer, int offset, int length)
      throws IOException {

      // create a byte array which will be used to read the datagram
      byte[] fullPkt = new byte[length];

      // the new Datagram
      Datagram packet = socket.newDatagram(fullPkt, length);

      // receive it
      socket.receive(packet);

      // extract the actual RTP Packet's media data in the requested buffer
      RTPPacket rtpPacket = getRTPPacket(packet, packet.getData());
      buffer = rtpPacket.getData();

      // debug
      System.err.println(rtpPacket + " with media length: " + buffer.length);

      // and return its length
      return buffer.length;
    }

    // extracts the RTP packet from each datagram packet received
    private RTPPacket getRTPPacket(Datagram packet, byte[] buf) {

      // SSRC
      long SSRC = 0;

        // the payload type
        byte PT = 0;

      // the time stamp
        int timeStamp = 0;

        // the sequence number of this packet
        short seqNo = 0;


        // see http://www.networksorcery.com/enp/protocol/rtp.htm
        // for detailed description of the packet and its data
        PT =
          (byte)((buf[1] & 0xff) & 0x7f);

        seqNo =
          (short)((buf[2] << 8) | ( buf[3] & 0xff));

        timeStamp =
          (((buf[4] & 0xff) << 24) | ((buf[5] & 0xff) << 16) |
            ((buf[6] & 0xff) << 8) | (buf[7] & 0xff)) ;

        SSRC =
          (((buf[8] & 0xff) << 24) | ((buf[9] & 0xff) << 16) |
            ((buf[10] & 0xff) << 8) | (buf[11] & 0xff));


        // create an RTPPacket based on these values
        RTPPacket rtpPkt = new RTPPacket();

        // the sequence number
        rtpPkt.setSequenceNumber(seqNo);

        // the timestamp
        rtpPkt.setTimeStamp(timeStamp);

        // the SSRC
        rtpPkt.setSSRC(SSRC);

        // the payload type
        rtpPkt.setPayloadType(PT);

        // the actual payload (the media data) is after the 12 byte header
        // which is constant
        byte payload[] = new byte [packet.getLength() - 12];

        for(int i=0; i < payload.length; i++) payload [i] = buf[i+12];

        // set the payload on the RTP Packet
        rtpPkt.setData(payload);

        // and return the payload
        return rtpPkt;

    }

    public long seek(long where) throws IOException {
     throw new IOException("cannot seek");
    }

    public long tell() { return -1; }

    public int getSeekType() { return NOT_SEEKABLE;    }

    public Control[] getControls() { return null; }

    public Control getControl(String controlType) { return null; }

    public long getContentLength() { return -1;    }

    public int getTransferSize() { return -1;    }

    public ContentDescriptor getContentDescriptor() {
        return new ContentDescriptor("audio/rtp");
    }
}

The constructor for the RTPSourceStream creates a SocketConnection to the remote server (hard-coded to the local server and port here, but you can change this to accept any server or port). It then opens the input and output streams, which it uses to create the RTSPProtocolHandler. Finally, using this handler, it sends the DESCRIBE and SETUP commands to the remote server to get the server ready to send the packets. The actual delivery doesn't start until the start() method is called by the StreamingDataSource, which opens up a local port (hard-coded to 8081 in this case) for receiving the packets and sends the PLAY command to start receiving these packets. The actual reading of the packets is done in the read() method, which receives the individual packets, strips them to create the RTPPacket instances (with the getRTPPacket() method), and returns the media data in the buffer supplied while calling the read() method.

A MIDlet to see if it works

With all the classes in place, let's write a simple MIDlet to first create a Player instance that will use the StreamingDataSource to connect to the server and then get media packets from it. The Player interface is defined by the MMAPI and allows you to control the playback (or recording) of media. Instances of this interface are created by using the Manager class from the MMAPI javax.microedition.media package (see the MMAPI tutorial). The following shows this rudimentary MIDlet:

import javax.microedition.media.Player;
import javax.microedition.midlet.MIDlet;
import javax.microedition.media.Manager;

public class StreamingMIDlet extends MIDlet {

  public void startApp() {

    try {

      // create Player instance, realize it and then try to start it
      Player player =
        Manager.createPlayer(
          new StreamingDataSource(
            "rtsp://localhost:554/sample_100kbit.mp4"));

      player.realize();

      player.start();

    } catch(Exception e) {
            e.printStackTrace();
    }
  }

  public void pauseApp() {}

  public void destroyApp(boolean unconditional) {}
}

So what should happen when you run this MIDlet in the Wireless toolkit? I have on purpose left out any code to display the resulting video on screen. When I run it in the toolkit, I know that I am receiving the packets because I see the debug statements as shown in Figure 2.

Figure 2. Running StreamingMIDlet output

The RTP packets as sent by the server are being received. The StreamingDataSource along with the RTSPProtocolHandler and RTPSourceStream are doing their job of making the streaming server send these packets. This is confirmed by looking at the streaming server's admin console as shown in Figure 3.

Figure 3. Darwin's admin console shows that the file is being streamed (click for full-size image).

Unfortunately, the player constructed by the Wireless toolkit is trying to read the entire content at one go. Even if I were to make a StreamingVideoControl, it will not display the video until it has read the whole file, therefore defeating the purpose of the streaming aspect of this whole experiment. So what needs to be done to achieve the full streaming experience?

Ideally, MMAPI should provide the means for developers to register the choice of Player for the playback of certain media. This is easily achieved by providing a new method in the Manager class for registering (or overriding) MIME types or protocols with developer-made Player instances. For example, let's say I create a Player instance that reads streaming data called StreamingMPEGPlayer. With the Manager class, I should be able to say Manager.registerPlayer("video/mpeg", StreamingMPEGPlayer.class) or Manager.registerPlayer("rtsp", StreamingMPEGPlayer.class). MMAPI should then simply load this developer-made Player instance and use this as the means to read data from the developer-made datasource.

In a nutshell, you need to be able to create an independent media player and register it as the choice of instance for playing the desired content. Unfortunately, this is not possible with the current MMAPI implementation, and this is the data consumption conundrum that I had talked about earlier.

Of course, if you can test this code in a toolkit that does not need to read the complete data before displaying it (or for audio files, playing them), then you have achieved the aim of streaming data using the existing MMAPI implementation.

This experiment should prove that you can stream data with the current MMAPI implementation, but you may not be able to manipulate it in a useful manner until you have better control over the Manager and Player instances. I look forward to your comments and experiments using this code.

Resources

• Sample code for this article
• Sun's MMAPI page
• Simple RTSP Client Steps required to create an RTSP client
• RTSP RFC and RTP RFC
• Apple's Darwin Stream Server information page
  
• Mobile Media API (MMAPI) Book
• Mobile Media API (MMAPI) Tutorial

Vikram Goyal is the author of Pro Java ME MMAPI.

<!-- end content -->

View all java.net Articles.

Despite its limitations, what do you think about MMAPI and its prospects for streaming?

<!-- rbcontent & rbox closed in cs_disc/thread_header.view -->

<!-- closing divs from a/print.view, et al -->

Showing messages 1 through 17 of 17.

• trouble with player
  2007-07-26 08:22:48neopogo [Reply | View]
  
  Hi Vikram
  
  In the begining want to apologize for english, it isn't good
  
  I have a question, is it possible to play dowloaded data stream in QuickTime Player (i mean foward downloaded packets to QT player) ? Because i have the same problem like other people with play movie in player defined by MMAPI.

• Splitting the Streams
  2007-05-24 08:52:43tomshark [Reply | View]
  
  Hi Vikram,
  
  The article was great; I happened upon it after waiting anxiously to see the video play on the phone only to have to wait for what seemed like several minutes for it to play; and it was only a 30-second clip. I thought ridicious, whose going to wait 3-5 minutes to see a 30-second clip? Then if you try to view a video on utube, the phone says you don't have enough memory to view the clip. We'll that just sucks; what good is the phone if I cannot view a video on utube; I don't want to just watch 'carrier' videos; I want to explore other content on the Internet. I just didn't understand why MobiTV is streaming but not me? Is there some type of hidden API that they are leasing from the carriers or what?
  Ok, back to the code - the splitting of the streams seems to work good for small clips - 30-seconds. Eventually, I'll fine tune the split to work with larger video clips; I'll have to split-up the audio into smaller chunks (keeping the small audio chunks in a buffer and rotate players (one player pretching while the other player is playing). Initial test results show that the music is jitterly on the sun toolkit 2.5; however, surprisingly, it plays fine on the phone. Since I haven't signed the midlet yet, I cannot do a full test (without the constant airtime confirmations). Other test results show that the video resolution is low on the toolkit, but improves greatly when viewing on the phone; I guess due to smaller phone screen (this should allow me to lower the resolution of the video resulting a quicker "stream"). I'm using a LG-CU400 for testing; previews of development code can be found by googling tomshark - warning content my be offensive. One of my first goals is to be able to loop through small video clips (15-60) seconds with minimum buffering time. Eventually, I work towards viewing larger clips; does anyone know if a .wav file can be broken up easily, if so, what are some tools (called via scripts) that can do the job?

• Possible bug in the code.
  2007-05-20 13:11:13tpanton [Reply | View]
  
  RTPSourceStream.read()
  assigns to the buffer variable - That's not what you
  want, you need to arraycopy the data into the buffer
  you were passed.
  Also you could be passed a buffer that's to small
  to hold both the data and the header info.
  
  Interesting ideas though.

• Just find a very good site
  2007-03-15 15:06:53drhu00 [Reply | View]
  
  Just find a very good site that implemented the jsr135. Open source project! Email me at [email protected] if you want this site's url.
  With this jsr135 open source, I think at least we can implement the rtsp protocol in SUN WTK25 emulator.

• Does this approach really works?
  2007-03-11 20:26:46drhu00 [Reply | View]
  
  I tested this application today on WTK25, Motorola Emulator, Sony Ericsson Emulator and Nokia Emulator. None of them are working ! I setup the Darwin servers and I can use realplay and quicktime play the sample_50kbit.3gp which means the rtsp server is ok. But when I tested in the emulator, either the read data hanging or the create player failed. Does anybody make it works? Does anybody ever tested it in the real mobile handset? I am just curious if this approach really works. We all know that J2ME Streaming is a hard topic these days.

• reference to code and article
  2007-03-06 05:11:06willmepham [Reply | View]
  
  Hello there Vikram. Thanks for the article and the code.
  
  I'm doing some research on media streaming in the J2ME and would like to be able to use some of the code you provide. Would it be ok for me to use and reference some of your code for a journal article? It will be attributed to you and there will be a full reference to this article.
  
  Thanks once again, Will

• 3gp streaming rtsp
  2007-02-12 04:54:58bunterelefant [Reply | View]
  
  hi vikram,
  
  thanks for the rtsp implementation. i used your code and noticed some points:
  
  have you ever tried to read the whole video file? after the end of the file is reached the read method in the RTSPSourceStream class just hangs. I changed it, so that it returns a -1 after a fixed amount of read data. result: after the 'end' of the stream is reached, the message "The data is not an MPEG stream." appears on the error output.
  i can imagine two reasons for that behaviour:
  
  - the server doesn't stream the whole file, because it doesn't get a feedback (rtcp) from the client, and stops streaming after one minute. so the player tries to parse the whole file, but the stream is corrupted, because of the missing end of the file.
  but i can't imagine, that the player refuses playing because of some missing bits at the end of the stream.
  
  -It also might be, that the player really can't handle the format in which the content is deliverd. I updated the RTPPacket class, so that it decodes the full rtp packet header. so i noticed, that the server is sending two different types of packets - one for video and one for audio. i can't imagine that the 3gp container seperates content so stricktly, that it can be sent in two packet. this means, that the delivered content isn't in a 3gp container anymore (correct me if i'm wrong). so there are some questions: in which format is the content? is the player capable of playing this format? it that the reason of the player for reading the whole stream?
  i also to tried to discard the audio packets, but it also failed.
  
  perhaps you have more background knowlege about streaming, than i have, so you know where i'm wrong.

• Can it be played in mobile
  2007-01-13 08:28:27thelastdon [Reply | View]
  
  Hi
  Suppose i am able to create a player and play the video in toolkit ,how is it possible to use the same in mobile i.e use the darwin server and active perl?

• stream video
  2007-01-03 16:15:09skylancer3id [Reply | View]
  
  hi vikram
  could teach me how to stream video from another handphone
  so, the video source from another handphone not in the res's folder like your code above
  please help me
  thanks vikram..

• I think problem is J2ME MAPI doesnt support mpeg4
  2006-11-29 09:23:05j2meid [Reply | View]
  
  From quick look at source code and sampel stream used for testing it looks like problem is because we are trying to stream mpeg4 media using Darwin and MMAPI doesnt support that.
  Have you tried with MPEG-1?
  
  Please correct me if you think I am wrong.
  
  J2ME User

• How the player gets Realized ?
  2006-11-12 04:23:42kamanashisroy [Reply | View]
  
  Hi,
  I have written a similar code for amr data. There I created a player from datasource instance. And there is a associated SourceStream too. After player creation when I want to realize, it reads from the stream. Like amr file , I give the player "#!AMR/n" as the first six bytes. The player.realize() does not return on this. It tries to read more. And after few tries it throws a MediaException as "Unable to Realize".
  Is it true that the realization processes is for acquiring all the amr data ? Is it possible to start the player before there is data available for it to play ? If possible Then how can I go realized, prefetched and started states without giving it the total data.

• realize() does´t end
  2006-11-05 12:00:33sustmi [Reply | View]
  
  Hello, thank for this example but:
  When I tries it in J2ME WTK from Sony Ericsson or in WTK22 it works good, but when realize is called it starts reading data, then it reads all data and realize doesn´t end.
  So no Player.start() is called.
  Where is the mistake?
  Thank you for helping and excuse my english, please :)

• How could it be run in Nokia SDK env?
  2006-10-10 09:50:05therock2006 [Reply | View]
  
  The sample is OK for WTK22. However when I try to edit it and let it work with Nokia S60 emulator, it shows error, which is as follow.
  1) video/mpeg in StreamingDataSource.java
  javax.microedition.media.MediaException: Cannot create a player for video/mpeg
  2) Then I changed video/mpeg to video/3gpp, then error is :
  MMA: Exception during remote invocation
  javax.microedition.media.MediaException.
  
  So I wonder how to let it work in Nokia SDK. Many thanks.

• Linking RTPPacket with Player
  2006-10-08 16:24:26arnob [Reply | View]
  
  Hello Vikram !
  Thanks a lot for your smart code. It works well for me (with toolkit and 3GPP Audio File and an other stream server).
  I wonder how can I "link" the RTP Packet with the Player instance for complete the code and ear the stream and get back the control to the GUI ?
  Does the PLAY at "somewhere in the stream" can work well with this principle (I think so !) ?
  Best regards,
  Arno

• ?
  2006-09-21 02:49:19ugmusic221 [Reply | View]
  
  Has anyone tryed to use this app on a working cell phone?

• http
  2006-09-06 19:32:24ztx [Reply | View]
  
  Hi,
  can it use http instead of socket

• problem when playing
  2006-08-29 08:59:28fredericuillandre [Reply | View]
  
  your article is really interesting.
  i just have a little problem with the app:
  when i launch the app, everything works fine. the problem is that the app freezes at the last server response:
  

  ====== SERVER RESPONSE ======
  RTSP/1.0 200 OK=
  Server: DSS/5.5 (Build/489.7; Platform/Linux; Release/Darwin; )
  Cseq: 4
  Session: 1758903497669535194
  Range: npt=0.00000-160.40000
  RTP-Info: url=rtsp://myDarwin/test/aq1.3gp/trackID=65536;seq=11282;rtptime=552272254,url=rtsp://myDarwin/test/aq1.3gp/trackID=65537;seq=15444;rtptime=1373101550

  any idea. thanks
  frederic