一、前言
自适应流切换属于多路流切换的方式中的一种,ExoPlayer作为MediaCodec使用的集大成者,不仅具备通过MergingMediaSource实现不同流的组合切换,同样也具备基于MGEG-DASH、HLS、smoothing-stream 协议的的自适应流切换。当然,在项目中每种方案的选型都要充分考虑团队条件。
主要区别如下:
MergingMediaSource 方式更适合团队人力有限,后台服务支持有限的情况,不需要在资源传输和编码上做更多的考虑,普通的CDN部署就可以,相比更加节省成本。而自适应流相对要求比较专业,对服务器的部署、资源分片、资源编码也是有一定要求的。 MergingMediaSource 方式可实现不同编码的流合并,而自适应流方面部分协议如HLS有较严格的要求,主要要求是ts分片的编码尽可能保持一致,这样做的目的是为尽可能实现MediaCodec的重复利用。当然,MergingMediaSource方式如果每路流的Format差别不大,视频解码器完全可以通过PPS、SPS或flush buffer 的方式实现MediaCodec利用,音频解码器也是可以通过输入特定字节特征实现MediaCodec复用。 在ExoPlayer中,MergingMediaSource 中的同一类型(视频类型、音频类型、字幕类型等)的数据,由于缺乏必要的码率参数, 无法将相似Format的Track数据合并为一组,因此使用的FixedTrackSelection对同一类型的资源,自然而然也不支持多路流的自动切换。而自适应流完成可以实现Format分组,最终创建AdaptiveTrackSelection 动态管理各路流。
Renderer渲染器:负责解码器的Format支持能力检测、解码器的注册、解码器的销毁、解码器的复用、采样数据读取、数据渲染或输出、丢帧、跳帧以及音画同步等工作。ExoPlayer支持Renderer的拆解、组合、关闭和启用,也支持自定义的解码器接入,比如通过SimpleDecoder实现FFMPEG对视频和音频的解码渲染。
MediaClock媒体时钟:负责音画同步、播放进度管理等工作。在ExoPlayer中国存在两种时钟,一种是独立时钟StandaloneMediaClock,另一种是通过音频Renderer实现的Audio Master模式的时钟。
Extractor 解封装器:负责将媒体资源中的每一路流的Moov信息、采样表、Format、采样数据(如SPS、PPS、各种帧数据)拆解出来,同时会对一些数据,便于Track和Format的选择以及码流切换。ExoPlayer内置了大量的解封装器,同样也支持自定义的Extractor来实现特定目的。当然,自适应流Format的解析一般是通过MediaSource去解析的,只有视频容器需要通过Extractor去解析。
Decoder解码器:负责解码采样数据,其中MediaCodec具备使用硬解和软解的能力的同时,而且支持渲染。
TrackSelector:多路流切换的核心类,负责通过Format检测实现TrackSelection的分组、以及TrackGroup和Renderer、TrackGroup和TrackSelection的配对工作。
DataSource 数据源:负责提供数据。
MediaSource 媒体源:在ExoPlayer中,得益于对从DataSource中抽象出了MediaSource,使得ExoPlayer在多路流管理方面更加灵活方便。
SeekPoint:在ExoPlayer中,SeekPoint 往往是IDR帧即将开始的位置。
FixedTrackSelection:固定流选择器,播放过程中TrackSelection保持固定,普通协议默认的TrackSelection,但也因为这个原因,使得其适用于MergingMediaSource方式的多路流切换。
AdptiveTrackSelection: 自适应流选择器,可以根据Bandwidth实现动态选择分片。当然,可以通过一些策略,实现用户自行的切换,类似bilibili的码流切换。
TrackGroup : 同一类型资源的Track Format 分组。
MapTrackInfo : 这个类实际上是一个Renderer和TrackGroup相关信息的集合类,主要保存Renderer能力信息和TrackGroup信息,某种程度上可以看到数据格式和Renderer全局信息。
selection分组原理:TrackSelector会通过SELECTION_ELIGIBILITY_FIXED、SELECTION_ELIGIBILITY_ADAPTIVE对资源进行分组,当然前提是各个Format之间能够相互兼容,具体兼容逻辑参考VideoTrackInfo类和AudioTrackInfo类。
Bandwidth:ExoPlayer中对网速检测的重要工具,检测结果用于AdaptiveTrackSelection进行分片选择。
DefaultMediaSourceFactory 用于实现DataSource转换为MediaSource的路由工厂,通过mineType和uri后缀识别出创建那种MediaSource。
在不同网速时自动切换为兼容当前bitrate的媒体流,匹配条件一般在自适应流的清单文件中就已经提前设定了,保证当前网络的bitrate大于清单协议中媒体流的最低bandWidth,就可以切换到指定的媒体流Track。
默认情况下,自适应流的切换不需要查找SeekPoint,而是通过选择下一个分片实现。 默认情况下,自适应流通过网速检测实现了分片切换。 从图上可知,每个分片的的播放时间和I帧的开始位置也需要做到严格对齐。
注意:之所以强调默认情况,一个重要的原因是ExopPlayer具备高度可扩展性,我们可以通过修改部分代码实现其他行为。
清单文件解析
建立Renderer与TrackGroup、Selection之间的映射关系
开始分片加载
网速带宽检测 与 AdaptiveTrackSelection 选择合适的分片
解码器复用或重启
完成切换
3.2.1.1 hls 协议清单文件
EXTM3U#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 1",AUTOSELECT=YES,DEFAULT=YESEXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 2",AUTOSELECT=NO,DEFAULT=NO,URI="alternate_audio_aac/prog_index.m3u8"#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",DEFAULT=YES,AUTOSELECT=YES,FORCED=NO,LANGUAGE="en",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/eng/prog_index.m3u8"EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="en",URI="subtitles/eng_forced/prog_index.m3u8"EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Français",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="fr",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/fra/prog_index.m3u8"EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Français (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="fr",URI="subtitles/fra_forced/prog_index.m3u8"EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Español",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="es",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/spa/prog_index.m3u8"EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Español (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="es",URI="subtitles/spa_forced/prog_index.m3u8"EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="日本語",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="ja",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/jpn/prog_index.m3u8"EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="日本語 (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="ja",URI="subtitles/jpn_forced/prog_index.m3u8"#EXT-X-STREAM-INF:BANDWIDTH=263851,CODECS="mp4a.40.2, avc1.4d400d",RESOLUTION=416x234,AUDIO="bipbop_audio",SUBTITLES="subs"gear1/prog_index.m3u8EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=28451,CODECS="avc1.4d400d",URI="gear1/iframe_index.m3u8"#EXT-X-STREAM-INF:BANDWIDTH=577610,CODECS="mp4a.40.2, avc1.4d401e",RESOLUTION=640x360,AUDIO="bipbop_audio",SUBTITLES="subs"gear2/prog_index.m3u8EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=181534,CODECS="avc1.4d401e",URI="gear2/iframe_index.m3u8"#EXT-X-STREAM-INF:BANDWIDTH=915905,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=960x540,AUDIO="bipbop_audio",SUBTITLES="subs"gear3/prog_index.m3u8EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=297056,CODECS="avc1.4d401f",URI="gear3/iframe_index.m3u8"#EXT-X-STREAM-INF:BANDWIDTH=1030138,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=1280x720,AUDIO="bipbop_audio",SUBTITLES="subs"gear4/prog_index.m3u8EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=339492,CODECS="avc1.4d401f",URI="gear4/iframe_index.m3u8"#EXT-X-STREAM-INF:BANDWIDTH=1924009,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=1920x1080,AUDIO="bipbop_audio",SUBTITLES="subs"gear5/prog_index.m3u8EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=669554,CODECS="avc1.4d401f",URI="gear5/iframe_index.m3u8"#EXT-X-STREAM-INF:BANDWIDTH=41457,CODECS="mp4a.40.2",AUDIO="bipbop_audio",SUBTITLES="subs"gear0/prog_index.m3u8
3.2.1.2 DASH协议清单文件
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>
两种协议的共同点中我们可以明确的发现又很多共同点:
bandWidth: 网络带宽,也就是下载的速率,但是在清单文件中一般表示的是支持该数据流的最低下载速率。
mimeType : 资源类型
codecs: 资源编码类型 width: 视频宽度 height: 视频高度
和其他协议的资源不同的是,由于使用清单文件的原因,基本可以实现在解封装之前就能获取到必要的Format信息。
解析时清单文件时,如果使用的是HLS协议,ExoPlayer内部利用HlsPlaylistParser类作为清单文件解析工具,如果是DASH则使用DashManifestParser解析清单,依次类推,smoothing-stream 使用SsmanifestParser进行将进行清单文件解析。
使用DefaultMediaSourceFactory创建对应的自适应流MediaSource,如HlsMediaSource、DashMediaSource、SsMediaSource。 自适应流MediaSource创建Parser解析工具,当然HlsMediaSource比较特殊,不直接持有Parser,而是通过DefaultHlsPlaylistTracker时线对Parser的调度。 创建Loader和ParsingLoadable,ParsingLoadable类似于Runnable,属于加载任务,在ParsingLoadable中可以实现边加载边解析。 对每组Track Format进行分组,使用TrackGroup保存分组信息。
protected final Pair< RendererConfiguration[], ExoTrackSelection[]>selectTracks(MappedTrackInfo mappedTrackInfo,int[][][] rendererFormatSupports,int[] rendererMixedMimeTypeAdaptationSupport,MediaPeriodId mediaPeriodId,Timeline timeline)throws ExoPlaybackException {Parameters parameters;synchronized (lock) {parameters = this.parameters;if (parameters.constrainAudioChannelCountToDeviceCapabilities&& Util.SDK_INT >= 32&& spatializer != null) {// Initialize the spatializer now so we can get a reference to the playback looper with// Looper.myLooper().spatializer.ensureInitialized(this, checkStateNotNull(Looper.myLooper()));}}int rendererCount = mappedTrackInfo.getRendererCount();// 建立Renderers 与 TrackGroup 映射,注意,我们前面提到的MappedTrackInfo,保存有Renderer和TrackGroup的相关信息// 这里也会尝试对不同TrackGroup Format进行依据兼容性进行合并分组到definition中ExoTrackSelection. Definition[] definitions =selectAllTracks(mappedTrackInfo,rendererFormatSupports,rendererMixedMimeTypeAdaptationSupport,parameters);//对筛选出来的defintions 二次过滤applyTrackSelectionOverrides(mappedTrackInfo, parameters, definitions);applyLegacyRendererOverrides(mappedTrackInfo, parameters, definitions);//三次过滤,对Renderer不能使用解除映关系// Disable renderers if needed.for (int i = 0; i < rendererCount; i++) {.TrackType int rendererType = mappedTrackInfo.getRendererType(i);if (parameters.getRendererDisabled(i)|| parameters.disabledTrackTypes.contains(rendererType)) {definitions[i] = null;}}//建立TrackGroup与Selection之间的映射关系,并且传入BandwidthMeter用于获取网速策略的数据ExoTrackSelection[] rendererTrackSelections =trackSelectionFactory.createTrackSelections(definitions, getBandwidthMeter(), mediaPeriodId, timeline);// Initialize the renderer configurations to the default configuration for all renderers with// selections, and null otherwise.//创建Renderer初始化配置RendererConfiguration[] rendererConfigurations = new RendererConfiguration[rendererCount];for (int i = 0; i < rendererCount; i++) {.TrackType int rendererType = mappedTrackInfo.getRendererType(i);boolean forceRendererDisabled =parameters.getRendererDisabled(i) || parameters.disabledTrackTypes.contains(rendererType);boolean rendererEnabled =!forceRendererDisabled&& (mappedTrackInfo.getRendererType(i) == C.TRACK_TYPE_NONE|| rendererTrackSelections[i] != null);rendererConfigurations[i] = rendererEnabled ? RendererConfiguration.DEFAULT : null;}// Configure audio and video renderers to use tunneling if appropriate.if (parameters.tunnelingEnabled) {//这种属于隧道渲染,具体流程好像是MediaCodec不负责解码,可以将未解码的数据输出到驱动层,由驱动层处理,如ac3音频数据直接输出到AudioTrack中,这方面资料太少,后续在研究。maybeConfigureRenderersForTunneling(mappedTrackInfo, rendererFormatSupports, rendererConfigurations, rendererTrackSelections);}return Pair.create(rendererConfigurations, rendererTrackSelections);}
在ExoPlayer中默认使用改工厂适配Selection,具体逻辑如下
public final ExoTrackSelection[] createTrackSelections(Definition[] definitions,BandwidthMeter bandwidthMeter,MediaPeriodId mediaPeriodId,Timeline timeline) {ImmutableList<ImmutableList<AdaptationCheckpoint>> adaptationCheckpoints =getAdaptationCheckpoints(definitions);ExoTrackSelection[] selections = new ExoTrackSelection[definitions.length];for (int i = 0; i < definitions.length; i++) {Definition definition = definitions[i];if (definition == null || definition.tracks.length == 0) {continue;}//遍历所有的definition,对分组中tracks 数量为1的创建FixedTrackSelection,对存在多个的创建AdaptiveTrackSeletionselections[i] =definition.tracks.length == 1? new FixedTrackSelection(definition.group,/* track= */ definition.tracks[0],/* type= */ definition.type): createAdaptiveTrackSelection(definition.group,definition.tracks,definition.type,bandwidthMeter,adaptationCheckpoints.get(i));}return selections;}
至此,Renderer 、TrackGroup、selection的映射关系建立完成。
那么这里有个疑问,如果利用MergingMediaSource合并多路流并修改参数,能否也实现AdaptiveTrackSelection,进入试下自适应能力?答案是否定的,因为MergingMediaSource合并的是完整的资源,在使用过程中并不会调用TrackSelection相关方法,当然ExoPlayer也没有实现资源的动态分片。
3.2.3 分片加载
DASH、HLS、Smoothing-Stream 加载分片的时候,单个分片都是用各自的实现的ChunkSource类,但是对于存在多个分片情况,ExoPlayer利用ChunkSampleStream和HlsSampleStreamWrapper将分片队列管理起来,核心方法是continueLoading中的实现,大致相同,这里以HlsSampleStreamWrapper的为参考。
public boolean continueLoading(long positionUs) {if (loadingFinished || loader.isLoading() || loader.hasFatalError()) {return false;}boolean pendingReset = isPendingReset();//获取资源队列List<BaseMediaChunk> chunkQueue;//获取上次加载资源时间位置long loadPositionUs;if (pendingReset) {chunkQueue = Collections.emptyList();loadPositionUs = pendingResetPositionUs;} else {chunkQueue = readOnlyMediaChunks;loadPositionUs = getLastMediaChunk().endTimeUs;}//获取下一个分片chunkSource.getNextChunk(positionUs, loadPositionUs, chunkQueue, nextChunkHolder);boolean endOfStream = nextChunkHolder.endOfStream;Chunk loadable = nextChunkHolder.chunk;nextChunkHolder.clear();if (endOfStream) {// 如果没有资源可以加载了,标记加载结束pendingResetPositionUs = C.TIME_UNSET;loadingFinished = true;return true;}if (loadable == null) {return false;}//下面逻辑是加载状态直接的判断loadingChunk = loadable;if (isMediaChunk(loadable)) {BaseMediaChunk mediaChunk = (BaseMediaChunk) loadable;if (pendingReset) {// Only set the queue start times if we're not seeking to a chunk boundary. If we are// seeking to a chunk boundary then we want the queue to pass through all of the samples in// the chunk. Doing this ensures we'll always output the keyframe at the start of the chunk,// even if its timestamp is slightly earlier than the advertised chunk start time.if (mediaChunk.startTimeUs != pendingResetPositionUs) {primarySampleQueue.setStartTimeUs(pendingResetPositionUs);for (SampleQueue embeddedSampleQueue : embeddedSampleQueues) {embeddedSampleQueue.setStartTimeUs(pendingResetPositionUs);}}pendingResetPositionUs = C.TIME_UNSET;}mediaChunk.init(chunkOutput);mediaChunks.add(mediaChunk);} else if (loadable instanceof InitializationChunk) {((InitializationChunk) loadable).init(chunkOutput);}long elapsedRealtimeMs =loader.startLoading(loadable, this, loadErrorHandlingPolicy.getMinimumLoadableRetryCount(loadable.type));mediaSourceEventDispatcher.loadStarted(new LoadEventInfo(loadable.loadTaskId, loadable.dataSpec, elapsedRealtimeMs),loadable.type,primaryTrackType,loadable.trackFormat,loadable.trackSelectionReason,loadable.trackSelectionData,loadable.startTimeUs,loadable.endTimeUs);return true;}
getNextChunk是continueLoading中的核心方法,继续看代码。
public final void getNextChunk(long playbackPositionUs,long loadPositionUs,List<? extends MediaChunk> queue,ChunkHolder out) {if (fatalError != null) {return;}StreamElement streamElement = manifest.streamElements[streamElementIndex];if (streamElement.chunkCount == 0) {// There aren't any chunks for us to load.//没有分片可以加载了out.endOfStream = !manifest.isLive;return;}int chunkIndex; //获取最后一次加载片段的索引if (queue.isEmpty()) {chunkIndex = streamElement.getChunkIndex(loadPositionUs);} else {chunkIndex =(int) (queue.get(queue.size() - 1).getNextChunkIndex() - currentManifestChunkOffset);if (chunkIndex < 0) {//这个片段不存在,说明加载片段不完整// This is before the first chunk in the current manifest.fatalError = new BehindLiveWindowException();return;}}if (chunkIndex >= streamElement.chunkCount) {// This is beyond the last chunk in the current manifest.//这个索引不合法,直接停止加载out.endOfStream = !manifest.isLive;return;}long bufferedDurationUs = loadPositionUs - playbackPositionUs;long timeToLiveEdgeUs = resolveTimeToLiveEdgeUs(playbackPositionUs);MediaChunkIterator[] chunkIterators = new MediaChunkIterator[trackSelection.length()];for (int i = 0; i < chunkIterators.length; i++) {int trackIndex = trackSelection.getIndexInTrackGroup(i);chunkIterators[i] = new StreamElementIterator(streamElement, trackIndex, chunkIndex);}//最关键的地方,这里会根据网速,筛选出下一个分片trackSelection.updateSelectedTrack(playbackPositionUs, bufferedDurationUs, timeToLiveEdgeUs, queue, chunkIterators);long chunkStartTimeUs = streamElement.getStartTimeUs(chunkIndex);long chunkEndTimeUs = chunkStartTimeUs + streamElement.getChunkDurationUs(chunkIndex);long chunkSeekTimeUs = queue.isEmpty() ? loadPositionUs : C.TIME_UNSET;int currentAbsoluteChunkIndex = chunkIndex + currentManifestChunkOffset;//这里是重点,获取选下一个需要加载的分片int trackSelectionIndex = trackSelection.getSelectedIndex();//获取分片解封装器ChunkExtractor chunkExtractor = chunkExtractors[trackSelectionIndex];int manifestTrackIndex = trackSelection.getIndexInTrackGroup(trackSelectionIndex);Uri uri = streamElement.buildRequestUri(manifestTrackIndex, chunkIndex);//绑定要加载的片段信息out.chunk =newMediaChunk(trackSelection.getSelectedFormat(),dataSource,uri,currentAbsoluteChunkIndex,chunkStartTimeUs,chunkEndTimeUs,chunkSeekTimeUs,trackSelection.getSelectionReason(),trackSelection.getSelectionData(),chunkExtractor);}
网速检测使用的默认的DefaultBandWidthMeter进行测试,具体原理是监控数据的某一段时间的下载流量,计算出平均网速。AdaptiveTrack Selection#updateSelectedTrack中会利用选择测量数据,重新计算使用哪个队列的数据。
int newSelectedIndex = determineIdealSelectedIndex(nowMs, chunkDurationUs);核心逻辑就是通过bitrate去做比较
@Overridepublic void updateSelectedTrack(long playbackPositionUs,long bufferedDurationUs,long availableDurationUs,List<? extends MediaChunk> queue,MediaChunkIterator[] mediaChunkIterators) {long nowMs = clock.elapsedRealtime();long chunkDurationUs = getNextChunkDurationUs(mediaChunkIterators, queue);// Make initial selectionif (reason == C.SELECTION_REASON_UNKNOWN) {reason = C.SELECTION_REASON_INITIAL;//初始化或者还没有选择过则直接一次性选择,不走兼容流程selectedIndex = determineIdealSelectedIndex(nowMs, chunkDurationUs);return;}int previousSelectedIndex = selectedIndex;@C.SelectionReason int previousReason = reason;int formatIndexOfPreviousChunk =queue.isEmpty() ? C.INDEX_UNSET : indexOf(Iterables.getLast(queue).trackFormat);if (formatIndexOfPreviousChunk != C.INDEX_UNSET) {previousSelectedIndex = formatIndexOfPreviousChunk;previousReason = Iterables.getLast(queue).trackSelectionReason;}//选择匹配bitrate的formatint newSelectedIndex = determineIdealSelectedIndex(nowMs, chunkDurationUs);//判断如果该format所在的Track不在黑名单中,则走兼容逻辑if (!isBlacklisted(previousSelectedIndex, nowMs)) {// Revert back to the previous selection if conditions are not suitable for switching.Format currentFormat = getFormat(previousSelectedIndex);Format selectedFormat = getFormat(newSelectedIndex);//注意:进入这个地方就是一个坑点,如果切码流失败,原因是这里通过一些条件又给重置回去了long minDurationForQualityIncreaseUs =minDurationForQualityIncreaseUs(availableDurationUs, chunkDurationUs);if (selectedFormat.bitrate > currentFormat.bitrate&& bufferedDurationUs < minDurationForQualityIncreaseUs) {// The selected track is a higher quality, but we have insufficient buffer to safely switch// up. Defer switching up for now.//如果选择的码流大于当前的,但是buffering的数据不够去安全的切换,因此还是选择当前TracknewSelectedIndex = previousSelectedIndex;} else if (selectedFormat.bitrate < currentFormat.bitrate&& bufferedDurationUs >= maxDurationForQualityDecreaseUs) {// The selected track is a lower quality, but we have sufficient buffer to defer switching// down for now.//选择的码流小于当前的,但是buffer数据是足够,不至于去切换newSelectedIndex = previousSelectedIndex;}}// If we adapted, update the trigger.//触发选择条件reason =newSelectedIndex == previousSelectedIndex ? previousReason : C.SELECTION_REASON_ADAPTIVE;selectedIndex = newSelectedIndex;}
上面提到2个坑点,bufferedDurationUs 大小影响切换,因此在项目中有必要规避此问题。
核心点determineIdealSelectedIndex,这里的逻辑就是获取当前带宽,然后匹配Selection中的采样队列
private int determineIdealSelectedIndex(long nowMs, long chunkDurationUs) {//获取带宽数据long effectiveBitrate = getAllocatedBandwidth(chunkDurationUs);int lowestBitrateAllowedIndex = 0;for (int i = 0; i < length; i++) {if (nowMs == Long.MIN_VALUE || !isBlacklisted(i, nowMs)) {Format format = getFormat(i);//获取小于当前网速的Formatif (canSelectFormat(format, format.bitrate, effectiveBitrate)) {return i;} else {lowestBitrateAllowedIndex = i;}}}return lowestBitrateAllowedIndex;}
带宽数据获取
private long getTotalAllocatableBandwidth(long chunkDurationUs) {//注意这里将带宽x0.7f,实际上比实际值偏小了,因此设置网速时一定要除以0.7flong cautiousBandwidthEstimate =(long) (bandwidthMeter.getBitrateEstimate() * bandwidthFraction);long timeToFirstByteEstimateUs = bandwidthMeter.getTimeToFirstByteEstimateUs();if (timeToFirstByteEstimateUs == C.TIME_UNSET || chunkDurationUs == C.TIME_UNSET) {//默认情况下回到这里,带宽除以当前播放的速度 (倍速)return (long) (cautiousBandwidthEstimate / playbackSpeed);}float availableTimeToLoadUs =max(chunkDurationUs / playbackSpeed - timeToFirstByteEstimateUs, 0);return (long) (cautiousBandwidthEstimate * availableTimeToLoadUs / chunkDurationUs);}
3.2.5 解码器的复用和重启
核心逻辑如下:
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>0
但是如何验证切换完成了,实际上是有回调的,参考下面接口实现。
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>1
四、实验
4.2 实验方法:
自动以AdaptiveTrackSelection#Factory或者自定义BandwidthMeter,这里我们选择后者,因为改动较小。
4.2.1实现QmBandwidthMeter
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>2
4.2.2 设置到Builder中
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>3
注意:这里设置缓冲控制类的前文已经说过, 分片选择存在坑点,由于bufferedDurationUs值过大,可能造成降码流失效, AdaptiveTrackSelection可能会重置会原来的selectionIndex为上一个分片Track,导致降码流失败,有必要做以下兼容。
4.2.3 以切换为下面的Hls分片为例子,实现从1920x1080 -> 640x360的切换
链接地址:https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>4
在起播后5s后设置带宽
【1】起播时设置带宽1924009/0.7f
【2】起播10s后设置带宽577610/0.7f
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>5
注意: 这里除以0.7f上面也说了,是因为带宽冗余机制造成的。
验证方法,实现下面的回调
<!--Generated with https://github.com/google/shaka-packager version 97fc982-release--><MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns:cenc="urn:mpeg:cenc:2013" minBufferTime="PT2S" type="static" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" mediaPresentationDuration="PT734S"><Period id="0"><AdaptationSet id="0" contentType="audio" lang="en"><Representation id="0" bandwidth="131596" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="44100"><AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/><BaseURL>tears_audio_eng.mp4</BaseURL><SegmentBase indexRange="745-1664" timescale="44100"><Initialization range="0-744"/></SegmentBase></Representation></AdaptationSet><AdaptationSet id="1" contentType="video" maxWidth="1920" maxHeight="856" frameRate="12288/512" par="38:17"><Representation id="1" bandwidth="769255" codecs="avc1.42c01e" mimeType="video/mp4" sar="852:857" width="320" height="142"><BaseURL>tears_h264_baseline_240p_800.mp4</BaseURL><SegmentBase indexRange="827-1602" timescale="12288"><Initialization range="0-826"/></SegmentBase></Representation><Representation id="2" bandwidth="1774254" codecs="avc1.4d401f" mimeType="video/mp4" sar="2242:2249" width="854" height="380"><BaseURL>tears_h264_main_480p_2000.mp4</BaseURL><SegmentBase indexRange="829-1604" timescale="12288"><Initialization range="0-828"/></SegmentBase></Representation><Representation id="3" bandwidth="7203938" codecs="avc1.4d4028" mimeType="video/mp4" sar="855:857" width="1280" height="570"><BaseURL>tears_h264_main_720p_8000.mp4</BaseURL><SegmentBase indexRange="830-1605" timescale="12288"><Initialization range="0-829"/></SegmentBase></Representation><Representation id="4" bandwidth="18316946" codecs="avc1.64002a" mimeType="video/mp4" sar="856:857" width="1920" height="856"><BaseURL>tears_h264_high_1080p_20000.mp4</BaseURL><SegmentBase indexRange="832-1607" timescale="12288"><Initialization range="0-831"/></SegmentBase></Representation></AdaptationSet></Period></MPD>1
符合预期,成功实现了降码流
五、总结
ExoPlayer不仅支持多路流合并方式切换,也支持自适应流切换,具备高度可定制化的能力,因此,对于体验要求较高的场景,可完全通过修改自适应流相关接口实现更加顺滑的多路流切换。
ExoPlayer自适应流切换如果要改造的为用户所能选择的方式,需要修改BandwidthMeter和AdaptiveTrackSelection的一些参数。还有就是分片长度、分片IDR帧对齐、编码格式一致(尽可能解码器复用)是需要考虑的因素,另外分片可能切换等待时间较长,也是需要注意的问题。
推荐站内搜索:最好用的开发软件、免费开源系统、渗透测试工具云盘下载、最新渗透测试资料、最新黑客工具下载……




还没有评论,来说两句吧...