HTTP Random Access and Live Content

HTTP Random Access and Live Content CableLabs

858 Coal Creek Circle Louisville CO 80027 pratt@acm.org

AT&T

Atlanta GA US barbara.stark@att.com

CableLabs

858 Coal Creek Circle Louisville CO 80027 d.thakore@cablelabs.com

Applications and Real-Time HTTP http range unit live aggregation To accommodate byte range requests for content that has data appended over time, this document defines semantics that allow a HTTP client and server to perform byte-range GET and HEAD requests that start at an arbitrary byte offset within the representation and ends at an indeterminate offset. Discussion of this draft takes place on the HTTPBIS working group mailing list (ietf-http-wg@w3.org), which is archived at . Working Group information can be found at ; source code and issues list for this draft can be found at .

Some Hypertext Transfer Protocol (HTTP) clients use byte-range requests (Range requests using the "bytes" Range Unit) to transfer select portions of large representations. And in some cases large representations require content to be continuously or periodically appended - such as representations consisting of live audio or video sources, blockchain databases, and log files. Clients cannot access the appended/live content using a Range request with the bytes range unit using the currently defined byte-range semantics without accepting performance or behavior sacrifices which are not acceptable for many applications. For instance, HTTP clients have the ability to access appended content on an indeterminate-length resource by transferring the entire representation from the beginning and continuing to read the appended content as it's made available. Obviously, this is highly inefficient for cases where the representation is large and only the most recently appended content is needed by the client. Alternatively, clients can also access appended content by sending periodic open-ended bytes Range requests using the last-known end byte position as the range start. Performing low-frequency periodic bytes Range requests in this fashion (polling) introduces latency since the client will necessarily be somewhat behind the aggregated content - mimicking the behavior (and latency) of segmented content representations such as HLS or MPEG-DASH. And while performing these Range requests at higher frequency can reduce this latency, it also incurs more processing overhead and HTTP exchanges as many of the requests will return no content - since content is usually aggregated in groups of bytes (e.g. a video frame, audio sample, block, or log entry). This document describes a usage model for range requests which enables efficient retrieval of representations that are appended to over time by using large values and associated semantics for communicating range end positions. This model allows representations to be progressively delivered by servers as new content is added. It also ensures compatibility with servers and intermediaries that don't support this technique.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

This document recommends a two-step process for accessing resources that have indeterminate length representations. Two steps are necessary because of limitations with the Range request header and the Content-Range response header fields. A server cannot know from a range request that a client wishes to receive a response that does not have a definite end. More critically, the header fields do not allow the server to signal that a resource has indeterminate length without also providing a fixed portion of the resource. A client first learns that the resource has a representation of indeterminate length by requesting a range of the resource. The server responds with the range that is available, but indicates that the length of the representation is unknown using the existing Content-Range syntax. See for details and examples. Once the client knows the resource has indeterminate length, it can request a range with a very large end position from the resource. The client chooses an explicit end value larger than can be transferred in the foreseeable term. A server which supports range requests of indeterminate length signals its understanding of the client's indeterminate range request by indicating that the range it is providing has a range end that exactly matches the client's requested range end rather than a range that is bounded by what is currently available. See for details.

Establishing if a representation is continuously aggregating ("live") and determining the randomly-accessible byte range can both be determined using the existing definition for an open-ended byte-range request. Specifically, defines a byte-range request of the form:

byte-range-spec = first-byte-pos "-" [ last-byte-pos ] which allows a client to send a HEAD request with a first-byte-pos and leave last-byte-pos absent. A server that receives a satisfiable byte-range request (with first-byte-pos smaller than the current representation length) may respond with a 206 status code (Partial Content) with a Content-Range header indicating the currently satisfiable byte range. For example:

HEAD /resource HTTP/1.1 Range: bytes=0- returns a response of the form:

HTTP/1.1 206 Partial Content Content-Range: bytes 0-1234567/* from the server indicating that (1) the complete representation length is unknown (via the "*" in place of the complete-length field) and (2) that only bytes 0-1234567 were accessable at the time the request was processed by the server. The client can infer from this response that bytes 0-1234567 of the representation can be requested and returned in a timely fashion (the bytes are immediately available).

Once a client has determined that a representation has an indeterminate length and established the byte range that can be accessed, it may want to perform a request with a start position within the randomly-accessible content range and an end position at an indefinite "live" point - a point where the byte-range GET request is fulfilled on-demand as the content is aggregated. For example, for a large video asset, a client may wish to start a content transfer from the video "key" frame immediately before the point of aggregation and continue the content transfer indefinitely as content is aggregated - in order to support low-latency startup of a live video stream. Unlike a byte-range Range request, a byte-range Content-Range response header cannot be "open ended", per :

byte-content-range = bytes-unit SP ( byte-range-resp / unsatisfied-range ) byte-range-resp = byte-range "/" ( complete-length / "*" ) byte-range = first-byte-pos "-" last-byte-pos unsatisfied-range = "*/" complete-length complete-length = 1*DIGIT Specifically, last-byte-pos is required in byte-range. So in order to preserve interoperability with existing HTTP clients, servers, proxies, and caches, this document proposes a mechanism for a client to indicate support for handling an indeterminate-length byte-range response, and a mechanism for a server to indicate if/when it's providing a indeterminate-length response. A client can indicate support for handling indeterminate-length byte-range responses by providing a Very Large Value for the last-byte-pos in the byte-range request. For example, a client can perform a byte-range GET request of the form:

GET /resource HTTP/1.1 Range: bytes=1230000-999999999999 where the last-byte-pos in the Request is much larger than the last-byte-pos returned in response to an open-ended byte-range HEAD request, as described above. In response, a server may indicate that it is supplying a continuously aggregating ("live") response by supplying the client request's last-byte-pos in the Content-Range response header. For example:

GET /resource HTTP/1.1 Range: bytes=1230000-999999999999 returns

HTTP/1.1 206 Partial Content Content-Range: bytes 1230000-999999999999/* from the server to indicate that the response will start at byte 1230000 and continues indefinitely to include all aggregated content, as it becomes available. A server that doesn't support or supply a continuously aggregating ("live") response will supply the currently satisfiable byte range, as it would with an open-ended byte request. For example:

GET /resource HTTP/1.1 Range: bytes=1230000-999999999999 will return

HTTP/1.1 206 Partial Content Content-Range: bytes 1230000-1234567/* from the server to indicate that the response will start at byte 1230000 and end at byte 1234567 and will not include any aggregated content. This is the response expected from a typical HTTP server - one that doesn't support byte-range requests on aggregating content. A client that doesn't receive a response indicating it is continuously aggregating must use other means to access aggregated content (e.g. periodic byte-range polling). A server that does return a continuously aggregating ("live") response should return data using chunked transfer coding and not provide a Content-Length header. A 0-length chunk indicates the end of the transfer, per section 4.1 of .

A client that wishes to only receive newly-aggregated portions of a resource (i.e., start at the "live" point), can use a HEAD request to learn what range the server has currently available and initiate an indeterminate-length transfer. For example:

HEAD /resource HTTP/1.1 Range: bytes=0- With the Content-Range response header indicating the (or ranges) available. For example:

206 Partial Content Content-Range: bytes 0-1234567/* The client can then issue a request for a range starting at the end value (using a very large value for the end of a range) and receive only new content.

GET /resource HTTP/1.1 Range: bytes=1234567-999999999999 with a server returning a Content-Range response indicating that an indeterminate-length response body will be provided

206 Partial Content Content-Range: bytes 1234567-999999999999/*

Some representations lend themselves to front-end content deletion in addition to aggregation. While still supporting random access, representations of this type have a portion at the beginning (the "0" end) of the randomly-accessible region that become inaccessible over time. Examples of this kind of representation would be an audio-video time-shift buffer or a rolling log file. For example a Range request containing:

HEAD /resource HTTP/1.1 Range: bytes=0- returns

206 Partial Content Content-Range: bytes 1000000-1234567/* indicating that the first 1000000 bytes were not accessible at the time the HEAD request was processed. Subsequent HEAD requests could return:

Content-Range: bytes 1000000-1234567/*

Content-Range: bytes 1010000-1244567/*

Content-Range: bytes 1020000-1254567/* Note though that the difference between the first-byte-pos and last-byte-pos need not be constant. The client could then follow-up with a GET Range request containing

GET /resource HTTP/1.1 Range: bytes=1020000-999999999999 with the server returning

206 Partial Content Content-Range: bytes 1020000-999999999999/* with the response body returning bytes 1020000-1254567 immediately and aggregated ("live") data being returned as the content is aggregated.

One potential issue with this recommendation is related to the use of very-large last-byte-pos values. Some client and server implementations may not be prepared to deal with byte position values of 2^^63 and beyond. So in applications where there's no expectation that the representation will ever exceed 2^^63, a value smaller than this value should be used as the Very Large last-byte-pos in a byte-seek request or content-range response. Also, some implementations (e.g. JavaScript-based clients and servers) are not able to represent all values beyond 2^^53. So similarly, if there's no expectation that a representation will ever exceed 2^^53 bytes, values smaller than this limit should be used for the last-byte-pos in byte-range requests.

&RFC2119; &RFC7230; &RFC7233; Hypertext Transfer Protocol (HTTP) Parameters IANA &RFC4234;

Mark Nottingham, Patrick McManus, Julian Reschke, Remy Lebeau, Rodger Combs, Thorsten Lohmar, Martin Thompson, Adrien de Croy, K. Morgan, Roy T. Fielding, Jeremy Poulter.