Two companies that contributed to this white paper—Adobe and Microsoft—shared thoughts on
their journey towards fMP4.
For more than a decade, that business centered on the Real-Time Media Protocol (RTMP) that Adobe launched back in 2001 with Flash Player 6. RTMP stood at the heart of Adobe Flash Media Server (FMS) and many de facto workflows sprung up around the protocol, although it was extended over the years to address encryption (RTMP-E) and peer-to-peer delivery (RTMFP).
One such enhancement to FMS was a precursor to Adobe’s foray into HTTP delivery. Back in December 2009, with the release of FMS 3.5.3, Adobe essentially decoupled the buffer from the connection, allowing semi-stateless connection between player and server and empowering Flash Player to continue play back even if the connection dropped. Developers can use ActionScript to
reconnect to Flash Player to FMS and, if reconnection occurs before the buffer empties, there is no perceived disruption in client playback, critical for mobile video delivery on intermittent networks.
Adobe saw opportunity to provide similar workflows for devices that did not use its Flash Player plug-in. In 2010, Adobe announced Adobe HTTP Dynamic Streaming (HDS), an HTTP-based adaptive streaming approach that built on fMP4 as its base protocol.
“This is a very different approach than our RTMP delivery with Adobe FMS, which uses normal FLV or MP4 (F4V) file formats,” said Kevin Towes, Adobe’s senior product manager for delivery. “The MP4-Fragment format [Adobe calls it F4F] adheres to the industry standard which is important for re-usability. The format also enabled a major requirement for media streaming – protection—
allowing you to deliver your live or pre-recorded content, leverage the caching devices, and still maintain control over your video assets.”
As part of the new Flash Media Server, announced in September 2011, Adobe enhanced support for both manifest (F4M) and fragment (F4F) file formats. Adobe uses an XML-based solution it calls F4M version 2, similar in concept to a variant playlist in what Adobe calls “set level manifests”.
“Workflows in RTMP have more than a decade of robustness behind them,” said Towes, “but the same workflows haven't been as easy in HTTP. Our investment into the HTTP workflows—bringing them on par with RTMP workflows—allows established FMS customers to maintain their sizable desktop/laptop viewer base while also pursuing new iPad and other iOS device revenue streams.”
Modifications to F4F allow real-time packaging, protection and features such as variant playlists to act on par with RTMP-based workflows. For encryption, FMS encrypts F4F fragments within FMS, using either a built-in Flash Access encryption module or a stand-alone Flash Access server.
Adobe emphasizes the point that fMP4 natively allows for network optimization and content encryption, but it also notes that different devices require different optimizing tweaks for delivery. Leveraging FMS for both desktop and mobile devices, including creation of HLS-based transport streams, Adobe feels it can supply content to every device at optimal resolutions and bandwidths. Seeking is one such area: given the ability to identify and deliver a byterange via fMP4, Adobe can bring its “seek” enhancement from RTMP directly into an HTTP delivery scenario. When enabled, seeking occurs first within the buffer, significantly reducing the load on the server compared to traditional back-and-forth seeks between client and server. The duration of the buffer can be dynamically changed, something not currently possibly in MPEG-2 TS solutions, to allow for pseudo-DVR functions such as instant replay or simple time-shifting options within the player. This also allows playback at different speeds and frame-accurate stepping from scene to scene. “We try to help broadcasters realize new revenue streams,” said Adobe’s Kevin Towes, “regardless of platform being delivering to. Where Flash Player is available, we'll use RTMP or HTTP Dynamic Streaming to deliver world-class quality of experience, encryption and rights management. With other platforms, such as iOS-based devices, we'll do an equally good job. Adobe also sees the benefit of standards-based approaches like the Common File Format, HTML5 and MPEG-DASH.”
Microsoft. Microsoft had a significant installed Windows Media® user base for well over ten years. Yet the company announced in late 2008 it was moving toward fragmented MP4 with the launch of Smooth Streaming, a first-to-market adoption of fMP4. Smooth Streaming combines Windows Server® and Silverlight® (or other Smooth Streaming clients) with XML and SMIL manifests.
Manifests have an *.ismv extension. Manifests indicate to an adaptive streaming client what bitrates are available in a file set, where fragments reside, and header parameters from each video file. Microsoft stores like-bitrate "chunks" of video as movie fragments in a single file, using timecode information stored in each movie fragment (moof) as an index for particular segments to download
and play back. Smooth Streaming devices download movie fragments containing either a video closed Group of Pictures (GOP) or an integral number of audio sync frames, via a single HTTP request. The player than select segments (typically 2-4 seconds long) from multiple streams encoded with different bitrates, resolutions, languages, etc.
What were Microsoft’s reasons for choosing fMP4 over ASF? The company notes five:
• MP4 is a lightweight container format with less overhead than ASF
• MP4 is easier to parse in managed code (.NET) than ASF
• MP4 is itself a widely used standard, making 3rd party adoption / support more straightforward
• MP4 was architected with H.264 video codec support in mind
• MP4 was designed to natively support payload fragmentation within the file
“The MPEG-4 fragmented format was able to do everything we needed,” wrote Ben Waggoner, principle program manager at Microsoft, “so it was simplest to just use that rather than make up something new. We're not trying to make up a new file format here; just take advantage of existing technologies in a novel way.”
“ASF was a great format for bit-pumping,” wrote Waggoner in 2009, “with a MIPS-per-Mbps ratio better than other formats of its era. However, it simply wasn't well set up to allow a byterange to encapsulate a fragment that could potentially be a single closed GOP. Since server hardware is so much more powerful these days and proxy caching so dramatically reduces the load on the origin server, going for a somewhat more complex server-side parsing methodology made good sense.”
Microsoft chose to go “all in” around the standard, including use of *.ismv and *.mp4 extensions. What have they done with Smooth Streaming since its announcement in 2008? Quite a lot, actually. Smooth Streaming has seen significant traction with Silverlight, but doesn’t require Silverlight to deliver adaptive bitrate HTTP streaming, as evidenced by Smooth Streaming clients such as the Comcast® XFINITY® TV App for iOS devices and Linux-based Netgem set-top boxes. Such non-Silverlight clients can be created using a Smooth Streaming client source-porting kit from Microsoft that enables playback on any device.
In addition, publication of a Smooth Streaming specification, called PIFF (protected interoperable
file format) has been adopted—along with Microsoft PlayReady® DRM technology—for use in new Netflix ready devices and applications, including the Apple iPad. Microsoft has also enabled several compelling scenarios—including advertising and alternate language tracks—on top of fMP4. In addition, the Microsoft ecosystem surrounding the Smooth Streaming format includes two key tools: Expression® Encoder for live- and on-demand Smooth Streaming encoding and IIS Media Services. The latter rides on top of the Microsoft Internet Information Services (IIS) web server that ships in every version of Windows Server 2008.
Another very interesting part of the workflow, that I’ve covered for several trade publications as well as the Workflowed blog, is IIS Transform Manager, a server-side transcoding and transforming tool. Transform Manager can convert a variety of formats, including segmented .mp4 files, to Smooth Streaming presentations. It can also convert Smooth Streaming content for use on Apple iOS devices, converting from fMP4 to HLS (MPEG-2 TS) as part of its standard workflow. Transform Manager scales out on multi-core HPC clusters, plus allows audits of all file copies/movement to improve monitoring/management control.
Microsoft is committed to supporting the standards-based approach, including the Common File Format (CFF), Common Encryption (CENC) and the MPEG-DASH adaptive delivery over HTTP.
their journey towards fMP4.
For more than a decade, that business centered on the Real-Time Media Protocol (RTMP) that Adobe launched back in 2001 with Flash Player 6. RTMP stood at the heart of Adobe Flash Media Server (FMS) and many de facto workflows sprung up around the protocol, although it was extended over the years to address encryption (RTMP-E) and peer-to-peer delivery (RTMFP).
One such enhancement to FMS was a precursor to Adobe’s foray into HTTP delivery. Back in December 2009, with the release of FMS 3.5.3, Adobe essentially decoupled the buffer from the connection, allowing semi-stateless connection between player and server and empowering Flash Player to continue play back even if the connection dropped. Developers can use ActionScript to
reconnect to Flash Player to FMS and, if reconnection occurs before the buffer empties, there is no perceived disruption in client playback, critical for mobile video delivery on intermittent networks.
Adobe saw opportunity to provide similar workflows for devices that did not use its Flash Player plug-in. In 2010, Adobe announced Adobe HTTP Dynamic Streaming (HDS), an HTTP-based adaptive streaming approach that built on fMP4 as its base protocol.
“This is a very different approach than our RTMP delivery with Adobe FMS, which uses normal FLV or MP4 (F4V) file formats,” said Kevin Towes, Adobe’s senior product manager for delivery. “The MP4-Fragment format [Adobe calls it F4F] adheres to the industry standard which is important for re-usability. The format also enabled a major requirement for media streaming – protection—
allowing you to deliver your live or pre-recorded content, leverage the caching devices, and still maintain control over your video assets.”
As part of the new Flash Media Server, announced in September 2011, Adobe enhanced support for both manifest (F4M) and fragment (F4F) file formats. Adobe uses an XML-based solution it calls F4M version 2, similar in concept to a variant playlist in what Adobe calls “set level manifests”.
“Workflows in RTMP have more than a decade of robustness behind them,” said Towes, “but the same workflows haven't been as easy in HTTP. Our investment into the HTTP workflows—bringing them on par with RTMP workflows—allows established FMS customers to maintain their sizable desktop/laptop viewer base while also pursuing new iPad and other iOS device revenue streams.”
Modifications to F4F allow real-time packaging, protection and features such as variant playlists to act on par with RTMP-based workflows. For encryption, FMS encrypts F4F fragments within FMS, using either a built-in Flash Access encryption module or a stand-alone Flash Access server.
Adobe emphasizes the point that fMP4 natively allows for network optimization and content encryption, but it also notes that different devices require different optimizing tweaks for delivery. Leveraging FMS for both desktop and mobile devices, including creation of HLS-based transport streams, Adobe feels it can supply content to every device at optimal resolutions and bandwidths. Seeking is one such area: given the ability to identify and deliver a byterange via fMP4, Adobe can bring its “seek” enhancement from RTMP directly into an HTTP delivery scenario. When enabled, seeking occurs first within the buffer, significantly reducing the load on the server compared to traditional back-and-forth seeks between client and server. The duration of the buffer can be dynamically changed, something not currently possibly in MPEG-2 TS solutions, to allow for pseudo-DVR functions such as instant replay or simple time-shifting options within the player. This also allows playback at different speeds and frame-accurate stepping from scene to scene. “We try to help broadcasters realize new revenue streams,” said Adobe’s Kevin Towes, “regardless of platform being delivering to. Where Flash Player is available, we'll use RTMP or HTTP Dynamic Streaming to deliver world-class quality of experience, encryption and rights management. With other platforms, such as iOS-based devices, we'll do an equally good job. Adobe also sees the benefit of standards-based approaches like the Common File Format, HTML5 and MPEG-DASH.”
Microsoft. Microsoft had a significant installed Windows Media® user base for well over ten years. Yet the company announced in late 2008 it was moving toward fragmented MP4 with the launch of Smooth Streaming, a first-to-market adoption of fMP4. Smooth Streaming combines Windows Server® and Silverlight® (or other Smooth Streaming clients) with XML and SMIL manifests.
Manifests have an *.ismv extension. Manifests indicate to an adaptive streaming client what bitrates are available in a file set, where fragments reside, and header parameters from each video file. Microsoft stores like-bitrate "chunks" of video as movie fragments in a single file, using timecode information stored in each movie fragment (moof) as an index for particular segments to download
and play back. Smooth Streaming devices download movie fragments containing either a video closed Group of Pictures (GOP) or an integral number of audio sync frames, via a single HTTP request. The player than select segments (typically 2-4 seconds long) from multiple streams encoded with different bitrates, resolutions, languages, etc.
What were Microsoft’s reasons for choosing fMP4 over ASF? The company notes five:
• MP4 is a lightweight container format with less overhead than ASF
• MP4 is easier to parse in managed code (.NET) than ASF
• MP4 is itself a widely used standard, making 3rd party adoption / support more straightforward
• MP4 was architected with H.264 video codec support in mind
• MP4 was designed to natively support payload fragmentation within the file
“The MPEG-4 fragmented format was able to do everything we needed,” wrote Ben Waggoner, principle program manager at Microsoft, “so it was simplest to just use that rather than make up something new. We're not trying to make up a new file format here; just take advantage of existing technologies in a novel way.”
“ASF was a great format for bit-pumping,” wrote Waggoner in 2009, “with a MIPS-per-Mbps ratio better than other formats of its era. However, it simply wasn't well set up to allow a byterange to encapsulate a fragment that could potentially be a single closed GOP. Since server hardware is so much more powerful these days and proxy caching so dramatically reduces the load on the origin server, going for a somewhat more complex server-side parsing methodology made good sense.”
Microsoft chose to go “all in” around the standard, including use of *.ismv and *.mp4 extensions. What have they done with Smooth Streaming since its announcement in 2008? Quite a lot, actually. Smooth Streaming has seen significant traction with Silverlight, but doesn’t require Silverlight to deliver adaptive bitrate HTTP streaming, as evidenced by Smooth Streaming clients such as the Comcast® XFINITY® TV App for iOS devices and Linux-based Netgem set-top boxes. Such non-Silverlight clients can be created using a Smooth Streaming client source-porting kit from Microsoft that enables playback on any device.
In addition, publication of a Smooth Streaming specification, called PIFF (protected interoperable
file format) has been adopted—along with Microsoft PlayReady® DRM technology—for use in new Netflix ready devices and applications, including the Apple iPad. Microsoft has also enabled several compelling scenarios—including advertising and alternate language tracks—on top of fMP4. In addition, the Microsoft ecosystem surrounding the Smooth Streaming format includes two key tools: Expression® Encoder for live- and on-demand Smooth Streaming encoding and IIS Media Services. The latter rides on top of the Microsoft Internet Information Services (IIS) web server that ships in every version of Windows Server 2008.
Another very interesting part of the workflow, that I’ve covered for several trade publications as well as the Workflowed blog, is IIS Transform Manager, a server-side transcoding and transforming tool. Transform Manager can convert a variety of formats, including segmented .mp4 files, to Smooth Streaming presentations. It can also convert Smooth Streaming content for use on Apple iOS devices, converting from fMP4 to HLS (MPEG-2 TS) as part of its standard workflow. Transform Manager scales out on multi-core HPC clusters, plus allows audits of all file copies/movement to improve monitoring/management control.
Microsoft is committed to supporting the standards-based approach, including the Common File Format (CFF), Common Encryption (CENC) and the MPEG-DASH adaptive delivery over HTTP.
