FLV (H.264 + AAC) support #828

jeoliva · 2015-09-29T15:22:22Z

FLV is not one of the most used containers right now, and of course not a priority for any player, but I though it could be useful adding it to Exoplayer for 2 reasons:

FLV was pretty used not too much time ago and still there are a lot of legacy video content libraries that are using this container. I have even seen a couple of reported issues asking for FLV support so there are some interest on it.
And second, and most important, a share knowledge reason. In other words, I think this is really useful to allow other developers to better understand how to build their own extractors for Exoplayer. FLV is one of the simplest and best documented video/audio container formats so it is easy going through the code, looking at the parser, flv extractor, FLV specification, and see what is the architecture behind Exoplayer extractors. MP4 type of containers are much more complex, and are a so hard way of starting coding or trying to understand extractors :).

Regarding the implementation, as title stated, this pull request is adding support for FLV container, and more specifically, when content within FLV is encoded using H.264 and AAC.

ojw28 · 2015-10-11T17:54:20Z

In general I'm not a fan of adding support for legacy container formats. If we take it, it's something we have to commit to maintaining. However, as you say, this does look pretty simple, and so it seems like a reasonable request!

I'll post a few initial comments now, and try and have a more in-depth look later this week. Thanks!

ojw28 · 2015-10-11T17:57:00Z

library/src/main/java/com/google/android/exoplayer/extractor/ExtractorSampleSource.java

@@ -146,6 +146,13 @@ public UnrecognizedInputFormatException(Extractor[] extractors) {
    } catch (ClassNotFoundException e) {
      // Extractor not found.
    }
+    try {


Please update the list in the javadoc at the top of this class to say that FLV is supported. Also, if seeking is not supported (see comment below) then update the sentence that says seeking in AAC and MPEG-TS is not supported, to include FLV too.

jeoliva · 2015-10-14T12:39:31Z

Taking a look to all these things.

jeoliva · 2015-10-15T22:35:35Z

Just incorporated suggested changes. Thanks @ojw28 for all your comments!

ojw28 · 2015-10-22T00:03:38Z

Sorry for the delay looking at this. Will try and get to it tomorrow!

ojw28 · 2015-10-22T14:24:26Z

library/src/main/java/com/google/android/exoplayer/extractor/ExtractorSampleSource.java

@@ -60,7 +60,7 @@
 * <li>MPEG TS ({@link com.google.android.exoplayer.extractor.ts.TsExtractor}</li>


This list should contain FLV.

jeoliva · 2015-10-22T21:53:53Z

Changes applied

winlinvip · 2015-10-26T09:50:59Z

FLV is useful in live streaming with low-latency, similar to RTMP for flash.
For example, most live streaming server supports RTMP/HLS, but HLS is large latency 10s+ while RTMP 1s+. Actually the RTMP is similar to HTTP-FLV and some server(for instance, srs) already supported the HTTP-FLV/RTMP.

winlinvip · 2015-10-26T09:56:18Z

@jeoliva Thanks for your work, I merged this pull request to my fork, 👍

winlinvip · 2015-10-26T10:06:30Z

library/src/main/java/com/google/android/exoplayer/extractor/flv/AudioTagPayloadReader.java

+      Pair<Integer, Integer> audioParams = CodecSpecificDataUtil.parseAacAudioSpecificConfig(
+          audioSpecificConfig);
+
+      MediaFormat mediaFormat = MediaFormat.createAudioFormat(MimeTypes.AUDIO_AAC,


Compile failed when merge to dev-1.5.1-rc：

git/ExoPlayer/library/src/main/java/com/google/android/exoplayer/extractor/flv/AudioTagPayloadReader.java
Error:(111, 44) error: method createAudioFormat in class MediaFormat cannot be applied to given types;
required: int,String,int,int,long,int,int,List<byte[]>,String
found: String,int,int,long,Integer,Integer,List<byte[]>,
reason: actual and formal argument lists differ in length

Seems change to the bellow is ok:

MediaFormat.createAudioFormat(MediaFormat.NO_VALUE, MimeTypes.AUDIO_AAC,

I fix it at: winlinvip@115aee8

Yes. I've merged this and fixed the break. Thanks @jeoliva / @winlinvip !

FLV (H.264 + AAC) support

ojw28 · 2015-10-26T10:45:30Z

I've also submitted some stylistic tweaks just to bring the code in line with the rest of the source: 950cc70

ojw28 · 2015-10-26T11:57:24Z

And some further changes. I was able to simplify ScriptTagPayloadReader somewhat. Please take a look, and let me know if I accidentally broke anything!

ojw28 · 2015-10-26T12:32:59Z

One final question: Where did the test link you've added come from? I want to know if it's fine to copy it across to our own hosting. Obviously the underlying content (big buck bunny) is fine for us to copy. Depending on who packaged it, we might need to re-package it ourselves to copy it over. Please advise. Thanks!

winlinvip · 2015-10-27T01:27:03Z

I test the flv, seems ok for vod stream, for example, it's ok for http://www.ossrs.net/mp4/source.200kbps.768x320.flv。
But failed to play a flv live stream, which is send by chunked encoding by SRS server, the demo stream is http://ossrs.net:8081/live/livestream.flv or link。
I will find out why, maybe ExoPlayer depends on the file Content-Length? So not support chunked http stream?

ojw28 · 2015-10-27T20:30:37Z

Thanks. for the info. I made some further changes to the extractor in 4422e8a based on an internal review. I've put a couple of comments in the old version of the code in that change that explain why the new state "skipping to next tag header" state is necessary :).

winlinvip · 2015-10-28T02:54:22Z

@ojw28 Thanks~ I will try to add hasVideo and hasAudio in header in SRS server.

winlinvip · 2015-10-28T03:16:49Z

Fixed in ossrs/srs@50a7b9c, set hasVideo and hasAudio tag in flv header.
It works! And the ExoPlayer can play the HTTP-FLV and HTTP-TS stream, both live and vod:

HTTP-FLV: http://ossrs.net:8081/live/livestream.flv
HTTP-TS: http://ossrs.net:8081/live/livestream.ts

chodison · 2015-10-28T03:24:11Z

But live time is not moving

winlinvip · 2015-10-28T03:33:18Z

@chodison Not understand you, what's time moving means? The timestamp in FLV stream is increasing, maybe the player not display it?

winlinvip · 2015-10-28T03:38:35Z

I see, you means the timeline of ExoPlayer, that's caused by the flv onMetaData packet:

String onMetaData
Object (20 items)
    Property 'duration' Number 210.7

Because the flv stream is generated by ffmpeg to publish a flv file, which contains the duration, I will remove this field for live stream.

chodison · 2015-10-28T03:39:36Z

The player‘s position is always zero. And Cycle two times, it don't work.

winlinvip · 2015-10-28T04:00:35Z

Fixed in ossrs/srs@ef00005, remove the duration of flv for live streaming.

chodison · 2015-11-02T07:04:23Z

@ojw28 For flv vod video, can not ExoPlayer to support seeking?

ojw28 · 2015-11-02T16:39:44Z

See @jeoliva's note on this thread above seeking: In fact, FLV container doesn't include any kind of information to help on implementing seeking.

If you want your media to be seekable you should use a more appropriate container format, like MP4.

winlinvip · 2015-11-03T01:49:36Z

@ojw28 @jeoliva @chodison There is some workaround for flv to support seek, the service which produce the flv file must inject all keyframe offset to the onMetaData script tag, then player can got the offset to seek to specified time. However, it's not the standard of flv, it's possible to implements by this workaround.

winlinvip · 2015-11-03T01:52:09Z

Please refer to https://github.com/simple-rtmp-server/srs/wiki/v2_EN_FlvVodStream.

chodison · 2015-11-03T02:42:59Z

@ojw28 why the ffplay player can seek flv vod video?

winlinvip · 2015-11-03T02:47:11Z

@chodison The ffplay generally play a local flv vod file, it can skip one tag by tag, it's ok for disk but not good for http flv vod stream. The network file always need a header which store the keyframe and offset map, similar to mp4 container.

chodison · 2015-11-03T03:49:29Z

@winlinvip My tested url is http flv vod stream.

winlinvip · 2015-11-03T04:12:36Z

@chodison Http flv vod stream is ok to seek, to skip one tag by tag, it's not efficient way however. The acceptable way for a big flv vod file to seek, is inject the keyframe offset to the onMetaData.

ojw28 · 2015-11-03T04:15:28Z

There's an alternative "accepted" way where the server will allow you to specify the seek position as, for example, a query parameter. In that case the server does the brute force tag-by-tag search (it's probably implemented to have built an index already) and serve you the media from the specified seek position.

The takeaway here, however is that (a) FLV does not support seek properly, (b) we don't intend to support any hacks to make seeking in FLV possible in ExoPlayer, unless they're external contributions and do not extend anywhere outside of the FLV extractor package, and (c) if you're knowingly packaging your own media as FLV, you should stop and choose a more appropriate container like MP4.

winlinvip · 2015-11-03T04:39:57Z

@ojw28 Aggree to use MP4 instead for http vod stream. 😄

chodison · 2015-11-23T06:50:26Z

@winlinvip @ojw28 flv vod stream server supports http start or http range to seek.

winlinvip · 2015-11-23T07:17:26Z

@chodison Yep, but need the player to finger it out where to seek, for example:

player.seek(10s)

The player must conver the 10s to file offset, for instance, 3421bytes, then player must request from the offset:

http://server/f.flv?start=3421

Or request by range whatever.

The question is: how to know the offset and seconds relationship? The flv container never describe it.

There is a workaround, to save the offset-seconds map to the onMetadata(the scirpt tag) of flv, which need the player to parse this tag -- this is exactly what @ojw28 said about the standard flv not support seek.

I aggree, should never use none-standard, especially the player.

chodison · 2015-11-23T07:30:24Z

@winlinvip http://server/f.flv?start=time or request by range, file offset is from he proportion of the position to the duration.
seekTime/flvDuration == position/flvFileLength.

winlinvip · 2015-11-23T10:13:05Z

@chodison seekTime/flvDuration != position/flvFileLength.

mondain · 2015-11-23T20:12:16Z

The offset and time relationships are often sent in the metadata response; depending upon your server's implementation of course.

For information on this, look here: http://h264.code-shop.com/trac/wiki/FlashPlayer

ojw28 · 2015-11-23T21:44:01Z

We have no plans to support seeking in FLV. All of the approaches mentioned would add complexity to the upstream package, and would not work in a consistent way across different DataSource implementations. If you want seeking, you should use a container format the properly supports it (or DASH, SmoothStreaming or HLS).

chodison · 2015-11-24T01:40:55Z

@winlinvip Estimation calculation. The flv vod stream server will return the key frame for closest position

winlinvip · 2015-11-24T02:43:08Z

@chodison It is possible to implement the flv stream seek, but not official standard for it, so I think @ojw28 is right to not support this feature and use HLS or DASH.

For the server to seek the closest position, not all server support it, that is why not official standard.

FLV Support - Added Audio and Script Data readers

8ddc735

ojw28 mentioned this pull request Oct 11, 2015

ExoPlayer does not support FLV #860

Closed

ojw28 reviewed Oct 11, 2015
View reviewed changes

ojw28 reviewed Oct 22, 2015
View reviewed changes

FLV Support - Added Video Reader and parsing improvements

3e36f52

winlinvip reviewed Oct 26, 2015
View reviewed changes

ojw28 added a commit that referenced this pull request Oct 26, 2015

Merge pull request #828 from jeoliva/flv-experiment

f133524

FLV (H.264 + AAC) support

ojw28 merged commit f133524 into google:dev Oct 26, 2015

winlinvip mentioned this pull request Oct 27, 2015

Low latency streaming with ExoPlayer? #623

Closed

winlinvip referenced this pull request in ossrs/srs Oct 28, 2015

for ExoPlayer, add av tag in flv header. 2.0.197

50a7b9c

winlinvip referenced this pull request in ossrs/srs Oct 28, 2015

for ExoPlayer, remove duration for live.

ef00005

google locked and limited conversation to collaborators Jun 28, 2017

ojw28 added the should merge label Aug 9, 2017

		@@ -60,7 +60,7 @@
		* <li>MPEG TS ({@link com.google.android.exoplayer.extractor.ts.TsExtractor}</li>

FLV (H.264 + AAC) support #828

FLV (H.264 + AAC) support #828

Conversation

jeoliva commented Sep 29, 2015

ojw28 commented Oct 11, 2015

ojw28 Oct 11, 2015

Choose a reason for hiding this comment

jeoliva commented Oct 14, 2015

jeoliva commented Oct 15, 2015

ojw28 commented Oct 22, 2015

ojw28 Oct 22, 2015

Choose a reason for hiding this comment

jeoliva commented Oct 22, 2015

winlinvip commented Oct 26, 2015

winlinvip commented Oct 26, 2015

winlinvip Oct 26, 2015

Choose a reason for hiding this comment

winlinvip Oct 26, 2015

Choose a reason for hiding this comment

winlinvip Oct 26, 2015

Choose a reason for hiding this comment

ojw28 Oct 26, 2015

Choose a reason for hiding this comment

ojw28 commented Oct 26, 2015

ojw28 commented Oct 26, 2015

ojw28 commented Oct 26, 2015

winlinvip commented Oct 27, 2015

ojw28 commented Oct 27, 2015

winlinvip commented Oct 28, 2015

winlinvip commented Oct 28, 2015

chodison commented Oct 28, 2015

winlinvip commented Oct 28, 2015

winlinvip commented Oct 28, 2015

chodison commented Oct 28, 2015

winlinvip commented Oct 28, 2015

chodison commented Nov 2, 2015

ojw28 commented Nov 2, 2015

winlinvip commented Nov 3, 2015

winlinvip commented Nov 3, 2015

chodison commented Nov 3, 2015

winlinvip commented Nov 3, 2015

chodison commented Nov 3, 2015

winlinvip commented Nov 3, 2015

ojw28 commented Nov 3, 2015

winlinvip commented Nov 3, 2015

chodison commented Nov 23, 2015

winlinvip commented Nov 23, 2015

chodison commented Nov 23, 2015

winlinvip commented Nov 23, 2015

mondain commented Nov 23, 2015

ojw28 commented Nov 23, 2015

chodison commented Nov 24, 2015

winlinvip commented Nov 24, 2015