State Of Play : Adaptive Streaming, the new kid in town

This is an installment of our ongoing State of Play series, designed to offer insight into new evolutions and significant issues in the online video industry.

In a previous article, we presented you with an overview of the main streaming protocols. In this edition, we take a closer look at the new technique of adaptive streaming (aka bitrate switching) for on-demand media. In a future edition, we will further examine support for live streaming and discover how a single broadcast can result in multiple simultaneously live streams, using different streaming protocols.

TECHNOLOGY OVERVIEW

Adaptive streaming is a technology in which the bitrate of a stream is dynamically adapted to the user's needs. The technique consists in detecting the user's bandwidth and CPU capacity in real time and adjusting the quality of the video stream accordingly. If the viewer's bandwidth or CPU capacity drops, the player can automatically switch to a lower resolution without interrupting the viewing experience.

Basically, all adaptive bitrate streaming technologies work in the same way. In advance, the source video file (or live event) must be encoded to multiple resolutions and/or data rates. Before delivery, these encoded files are divided into segments - sometimes referred to as "chunks" - each representing two to ten seconds of video. When the client (= video player) first contacts the streaming server, it receives a table of URLs in which each URL points to a specific time interval (the columns) of a specific quality (the rows). A client then starts requesting segments from the Web server, downloading them via HTTP. As the segments are downloaded to the client, it plays back the segments in the order received. While playing, the client monitors playback-related indicators such as file download time, buffer levels, and CPU utilization. If desirable, the client can switch to a higher or lower quality stream, which should result in a seamless user experience.

MULTIPLE PROTOCOLS

One issue with Adaptive Streaming is that implementations were not born out of a widely accepted standard, but instead every vendor has defined its own proprietary protocol. Recently, the Motion Pictures Experts Group (MPEG) has started work on a common standard called DASH, but this still has a long way to go.

Currently, there are three main flavors of HTTP adaptive streaming - by Apple, Microsoft and Adobe - each targeting their own platform. All three specifications are codec agnostic in theory, but the current implementations are limited to certain video and audio codecs. The delivery mechanisms have a lot in common, but are nevertheless incompatible; for instance, each protocol provides the table of available URLs to the client in a different fashion.

Apple HTTP Adaptive Streaming

• Available on iPhone, iPad, and iPod touch (requires iOS version 3.0 or later) or any device with QuickTime X or later installed.
• The current implementation requires H.264 video streams and AAC or MP3 audio streams which are stored inside a MPEG-2 Transport Stream (MPEG-2 TS) container.
• The streaming server communicates the available streams and their URLs to the client by exposing an m3u playlist (.m3u8).

Microsoft Smooth Streaming

• Supported by Microsoft's Silverlight framework, which is also included on Windows Phone 7.
• The current Silverlight framework requires the H.264 or VC-1 video codec, and the AAC or WMA audio codec. The container format is “fragmented” MPEG-4, a variant of the ISO specification in which not all the data in a regular MP4 file is included.
• The streaming server sends an XML-based .ismc manifest back to the client, which describes the available streams and their URLs.

Adobe HTTP Dynamic Streaming

• Supported by Flash Player 10.1 (or higher) and Adobe AIR 2.
• The Flash player currently supports H.264 or VP6 video and AAC or MP3 audio. The container format can either be FLV or “fragmented” MPEG-4.
• The streaming server sends an XML-based .F4M manifest back to the client, which describes the available streams and their URLs.

Adobe RTMP Dynamic Streaming

Before Adobe released their version of HTTP Dynamic Streaming, they had already included support in Flash Player (version 10 or higher) for adaptive streaming over RTMP. From the client's perspective, the bitrate switching mechanism is similar. The major difference is that RTMP is a stateful protocol with a persistent client-server connection. See our overview of streaming protocols for the benefits / drawbacks of using RTMP instead of HTTP.

BENEFITS

Adaptive streaming is particularly well suited for internet devices, which have different requirements regarding memory and processing power. Connections may vary from high-end (e.g. broadband TV) to low-end (e.g. 3G networks) and are often fluctuating. Adaptive streaming allows these devices to dynamically choose a stream with a suitable resolution and bitrate.

To get a fast start-up time, clients will usually begin by downloading the lowest available quality. During this first download, the client is able to calculate the available network bandwidth. The client then uses this knowledge to switch to the highest available quality, considering the available network bandwidth and processing power. Clients should also be able to quickly jump to a different part of the video by calculating the time interval and requesting the corresponding segment.

From an end-user's point of view, HTTP adaptive streaming is fully transparent. The client just needs the URL that points to the playlist or manifest file (which will be different for each flavor of HTTP streaming). After making its initial request to this URL, the client retrieves all necessary data to dynamically stream the video.

For RTMP adaptive streaming, some additional client-side work is needed. This set of articles by 'Adobe Developer Connection' covers the details of how to achieve dynamic streaming in Flash. However, embeddable video players like JW Player and Flowplayer have already implemented the bulk of these mechanisms and only require some basic client-side configuration.

Should You Consider Adaptive streaming ?

As more and more internet devices appear on the market with greater capabilities for video browsing and playback, adaptive streaming is starting to become a mainstream technology. However, it is still a young technology and a lot of its (added) value depends on the intelligence that is built into client applications. Although some recent studies have shown that clients can react quite effectively to bandwidth variations, player implementations are also said to be "still at their infancy“, which in some cases may lead to heterogeneous user experiences.

In our own experience HTTP adaptive streaming generally works well, although there are some differences :

• Apple's adaptive streaming looks very solid. It switches transparently from one bitrate to another and is capable of selecting the most suitable stream in different circumstances. Moreover, the Apple client devices can rely on hardware acceleration support when rendering video, which allows them to playback high-quality video's more easily.
• Adobe's HTTP dynamic streaming still has some limitations. These are caused by the overhead of the Flash platform, and the lack of hardware acceleration support in some mobile devices (we'll elaborate on these issues in a future edition of 'state of play'). This makes it sometimes more difficult for the client to choose the most suitable stream. Another consequence is that a buffering delay may occur when the client has to switch back to a lower bitrate. This being said, the average Flash client performs well under sufficient bandwidth conditions and most modern devices are capable of rendering decent quality video.
• The Microsoft Smooth Streaming player is quite effective under sufficient and/or stable bandwidth conditions. After having made a connection, it quickly tries to build a large playback buffer at the highest possible bitrate: this way, it can continue to play video at the same bitrate for quite some time after the connection has deteriorated. Once this buffer is exhausted however, the player may experience difficulties when switching back to a lower bitrate. This may result in buffering delay(s) or unnecessary bitrate reductions.

Anyhow, player implementations are rapidly getting better and - certainly as far as mobile devices are concerned - HTTP adaptive streaming seems to be the technology of the future. Should you decide to use it, Rambla takes care of the deployment issues for you. We have automated the whole workflow process of transcoding your files into multiple resolutions/bitrates and creating the necessary configuration files. All you have to do is upload your source file via FTP to a so-called 'hotfolder' and wait for the notification that provides you with your URL's for adaptive streaming. A default hotfolder is now available for every user account, free of charge. For developers, our RATS API includes the same functionality.