Persisten Connections, Pipelining and chunking
The Content-Length header is an interesting HTTP response header. This header tells HTTP client applications the size of the response. However, in HTTP 1.1, this header is optional. HTTP applications need not set this header if the size is not known. This is in fact the case with most applications that generate dynamic content.
When the Content-Length header is absent, how will HTTP clients be able to know how much to read? One obvious answer is to read till the end of the stream is reached, as in
But this will not work (without some additional work at a lower level) in this simple form when combined with two other features of HTTP 1.1, viz. persistent connections, and pipelining.
Persistent Connections
By default, HTTP 1.1 connections are persistent. HTTP servers do not close the connection immediately after serving the response, thus allowing the client to reuse the connection. Persistent connections are very useful when a client is fetching several resources from the same server over a short amount of time. Instead of creating a new connection for each resource, the client can reuse a single connection for all resources.
When persistent connections are in use, there are three ways a connection can be closed:
- The server can return a header Connection: close. This header indicates that the client should not assume that the connection is persistent after the current request is completed.
- The client can send a header Connection: close if it does not want to maintain the connection after the current request is completed.
- Timeout, or network failures.
If the client were to wait for each response before sending another request over a persistent connection, we can ignore the Content-Length header and try to read till the end of the stream. But it isn’t so, because connections can be pipelined.
Pipelining
With HTTP 1.1, the client need not wait for the current request to
complete before sending the next request over a persistent connection. It can send several requests over the same connection without waiting for responses for previous requests. This is called pipelining.
Here is a sample with three requests pipelined over a persistent connection.
GET / HTTP/1.1 Host: www.subbu.org User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 GET /counter.gif HTTP/1.1 Host: www.subbu.org User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 GET /styles.css HTTP/1.1 Host: www.subbu.org User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
The client can send several requests in sequence, and the server is supposed to send a response to each request in the same order the request was received. For the above requests, here is a sample set of responses from the server:
HTTP/1.1 200 OK
Date: Mon, 29 Nov 2004 01:45:35 GMT
Server: Apache
Last-Modified: Thu, 25 Nov 2004 02:32:28 GMT
Content-Length: 16291
Content-Type: text/html
<html>
<!-- Actual content snipped -->
</html>
HTTP/1.1 200 OK
Date: Mon, 29 Nov 2004 01:45:36 GMT
Transfer-Encoding: chunked
Content-Type: text/gif
255
[some bytes here]
104
[some bytes here]
0
HTTP/1.1 200 OK
Date: Mon, 29 Nov 2004 01:45:36 GMT
Transfer-Encoding: chunked
Content-Length: 2341
Content-Type: text/css
body {
# Actual content snipped
The server included a Content-Length header for the first and third requests, and therefore the client knows the number of bytes to read for these requests. But the server did not include a Content-Length header for the second response as the resource was dynamically generated.
Since the server did not include a
Content-Length header for the second response, the while loop to read all the available bytes fails for the second request. A client using such a while loop to read the stream till the end would incorrectly include the third response (and all subsequent responses) in the second response.
HTTP 1.1 solves this problem by chunking.
Chunking
When a HTTP server generates response dynamically, and does not include the Content-Length header, it includes a Transfer-Encoding: chunked header, and sends the response in chunks with known sizes, followed by a chunk of zero size. In the above example, the second response had three chunks. The last chunk in this response indicates the end of the response. With the help of the size of each chunk, and the delimiter chunk at the end, the client can safely determine the end of the response.
Most of the times, these details are automatically taken care by HTTP APIs. Client applications can safely try to read till the end of the stream. For example, the above while loop would work even when the connection is persistent and the server is using chunked response encoding. The underlying connection API (in this case, Java networking APIs) would merge the chunks and return -1 when the last chunk (of size 0) is reached. But if you are writing lower-level HTTP applications, or if a particular API does not support chunking, you must account for chunking explicitly.
Some HTTP applications incorrectly assume that the Content-Length header is required, and determine the length by buffering the response data in memory. But this is expensive and unnecessary. It is valid to skip this header and rely on chunking if the size is not known. But if the underlying network API does not support chunking, applications must make sure to send a Connection: close header so that the connection will not be persistent after the current response. Otherwise, clients supporting persistent connections may misbehave.