Until now, the demuxer cache was limited to a single range. Extend this
to multiple range. Should be useful for slow network streams.
This commit changes a lot in the internal demuxer cache logic, so
there's a lot of room for bugs and regressions. The logic without
demuxer cache is mostly untouched, but also involved with the code
changes. Or in other words, this commit probably fucks up shit.
There are two things which makes multiple cached ranges rather hard:
1. the need to resume the demuxer at the end of a cached range when
seeking to it
2. joining two adjacent ranges when the lowe range "grows" into it (and
resuming the demuxer at the end of the new joined range)
"Resuming" the demuxer means that we perform a low level seek to the end
of a cached range, and properly append new packets to it, without adding
packets multiple times or creating holes due to missing packets.
Since audio and video never line up exactly, there is no clean "cut"
possible, at which you could resume the demuxer cleanly (for 1.) or
which you could use to detect that two ranges are perfectly adjacent
(for 2.). The way how the demuxer interleaves multiple streams is also
unpredictable. Typically you will have to expect that it randomly allows
one of the streams to be ahead by a bit, and so on.
To deal with this, we have heuristics in place to detect when one packet
equals or is "behind" a packet that was demuxed earlier. We reuse the
refresh seek logic (used to "reread" packets into the demuxer cache when
enabling a track), which checks for certain packet invariants.
Currently, it observes whether either the raw packet position, or the
packet DTS is strictly monotonically increasing. If none of them are
true, we discard old ranges when creating a new one.
This heavily depends on the file format and the demuxer behavior. For
example, not all file formats have DTS, and the packet position can be
unset due to libavformat not always setting it (e.g. when parsers are
used).
At the same time, we must deal with all the complicated state used to
track prefetching and seek ranges. In some complicated corner cases, we
just give up and discard other seek ranges, even if the previously
mentioned packet invariants are fulfilled.
To handle joining, we're being particularly dumb, and require a small
overlap to be confident that two ranges join perfectly. (This could be
done incrementally with as little overlap as 1 packet, but corner cases
would eat us: each stream needs to be joined separately, and the cache
pruning logic could remove overlapping packets for other streams again.)
Another restriction is that switching the cached range will always
trigger an asynchronous low level seek to resume demuxing at the new
range. Some users might find this annoying.
Dealing with interleaved subtitles is not fully handled yet. It will
clamp the seekable range to where subtitle packets are.
libavcodec can't deal with them, because its API doesn't distinguish
between 0 sized packets and sending EOF. As such, keeping them doesn't
do any good, ever. This actually fixes some obscure mkv sample (see
previous commit).
Fixes some obscure sample that uses fixed size laces with 0-sized lace
size. Some broken shit. (Maybe the decoder wouldn't care about these
packets, but the demuxer attempted to resync after these packet reading
errors, even though they were perfectly recoverable. But I don't care
enough about this.)
Sample link: https://samples.ffmpeg.org/Matroska/switzler084d_dl.mkv
This directly reads individual mkv sub-packets (block laces) into a
dedicated AVBufferRefs, which can be directly used for creating packets
without a additional copy of the packet data. This also means we switch
parsing of block header fields and lacing metadata to read directly from
the stream, instead of a memory buffer.
This could have been much easier if libavcodec didn't require padding
the packet data with zero bytes. We could just have each packet
reference a slice of the block data. But as it is, the only way to get
padding without a copy is to read the laces into individually allocated
(and padded) memory block, which required a larger rewrite.
This probably makes recovering from broken mkv files slightly worse if
the transport is unseekable. We just read, and then check if we've
overread. But I think that shouldn't be a real concern.
No actual measureable performance change. Potential for some
regressions, as this is quite intrusive, and touches weird obscure shit
like mkv lacing. Still keeping it because I like how it removes some
redundant EBML parsing functions.
This adds a bunch of stuff (mostly unused or redundant) as preparation
for supporting multiple seek ranges. Actual support is probably still
far away.
One change that messes deeper with the actual code is that we account
for total buffered bytes instead of just the backwards bytes now. This
way, prune_old_packets() doesn't have to iterate over all seek ranges to
determine whether something needs pruning.
The main purpose of this commit is avoiding any hidden O(n^2) algorithms
in the code for pruning the demuxer cache, and for determining the
seekable boundaries of the cache. The old code could loop over the whole
packet queue on every packet pruned in certain corner cases.
There are two ways how to reach the goal:
1) commit a cardinal sin
2) do everything incrementally
The cardinal sin is adding an extra field to demux_packet, which caches
the determined seekable range for a keyframe range. demux_packet is a
rather general data structure and thus shouldn't have any fields that
are not inherent to its use, and are only needed as an implementation
detail of code using it. But what are you gonna do, sue me?
In the future, demux.c might have its own packet struct though. Then the
other existing cardinal sin (the "next" field, from MPlayer times) could
be removed as well.
This commit also changes slightly how the seek end is determined. There
is a note on the manpage in case anyone finds the new behavior
confusing. It's somewhat cleaner and might be needed for supporting
multiple ranges (although that's unclear).
The demuxer cache seeking mechanism looks at keyframe ranges to
determine the earlierst PTS of a packet. Instead of looping over all
packets twice (once to find the next keyframe, a second time to find the
seek PTS), do it in one go.
For that basically turn recompute_keyframe_target_pts() into an
iteration functionn. Functionality should be unchanged with this commit.
The base_ts field is used to guess the decoder position, and when set to
NOPTS, it just read ahead arbitrarily. Also demux_add_packet() sets
base_ts to the new timestamp when appending a packet, which would also
make it readahead by a too large amount.
Fix this by setting base_ts after a seek. This assumes that normally, a
cached seek target will always have the timestamp set. This is actually
not quite clear (as it calls recompute_keyframe_target_pts(), which
looks at multiple packets), but maybe it works well enough.
Don't do any of the extra work related to pruning the backbuffer if
demuxer cache seeking is disabled.
As a small extra, always prune packets with no timestamps immediately,
or queue heads that are not keyframes. (Both of them would be pruned
from the backbuffer by the normal logic anyway.)
If fulfilling --demuxer-readahead-secs requires more memory than allowed
by --demuxer-max-bytes, the queue obviously overflows. But the warning
is normally only for the case when trying to find the next video or
audio packet fails, and decoding can't continue.
Use a separate variable for determining whether to prefetch, and if the
queue has overflown, skip the message. In fact, skip the EOF setting and
waking up the decoder thread as well, because the EOF flag should not be
(have been) set in this situation, and waking up the reader thread helps
only if the EOF state changed.
In a shit show of subtle corner case interactions, making the demuxer
cache buffer the entire file can display a small buffered time if
subtitles are enabled. The reason is that some subtitle decoders may
read in advance infinitely, i.e. they read the entire subtitle stream.
Then, since the other streams (audio/video) have logically reached EOF,
and the subtitle stream is set to ds->active==true. This will have to be
fixed properly later to account buffering for subtitle-only files
(another corner case) correctly, but for now this is less annoying.
We don't hope to auto-detect them at load time, as that would be too
much of a pain - even FFmpeg requires fetching and parsing of video
packets, and exposes the information only via deprecated API.
But there still needs to be a way to select them by default. This is
also needed to get the first CC packet at all (without seeking back).
This commit also attempts to clean up locking a bit, which is a PITA,
but it's better be careful & clean.
Even though only 1 seek range is supported at the time.
Other than preparation for possibly future features, the main gain is
actually that we finally separate the reporting for the buffering, and
the seek ranges. These can be subtly different, so it's good to have a
clear separation.
This commit also fixes that the ts_reader wasn't rebased to the start
time, which could make the player show "???" for buffered cache amount
in some .ts files and others (especially at the end, when ts_reader
could become higher than ts_max). It also fixes writing the cache-end
field in the demuxer-cache-state property: it checked ts_start against
NOPTS, which makes no sense.
ts_start was never used (except for the bug mentioned above), so get rid
of it completely. This also makes it convenient to move the segment
check for last_ts to the demux_add_packet() function.
It isn't all that reliable, and improving it would make startup slower
and require more complexity. There isn't even a good reason to do this
(other than semi-broken mkv files), so don't do it. Also see previous
commit.
Avoids that cache seeking is not possible with files that have audio
frames with no timestamps (such as avi, sometimes mkv sub-packets from
lacing). These would set back_pts (first seekable PTS) to NOPTS, and
thus disable cache seeking completely. Instead, prune such packets until
we find one with timestamps.
One corner case is that the new next good packet might be in the forward
cache. In this case we defer dropping until the next time this code is
run, and the reader position has possibly moved past the drop point.
In theory, start/ts_min could be set to NOPTS, in which case
"pts < start" for a valid pts would always evaluate to false.
Also remove the redundant "in-cache seek is possible.." message, as
there's always another message when cache seeks are done.
The seek range computation ignored streams with no timestamps. For
things like buffer estimation this is OK and wanted, but the seek range
needs to be conservative.
Which parts of the queue are considered forward or backward cache
depends on ds->reader_header. The packet at ds->reader_head, as well as
all packets following it (via the ->next field) are considered forward.
The fw_packs/fw_bytes/bw_bytes must be updated accordingly.
This broke in demux_add_packet(), when ds->reader_head was _not_ set on
the first packet (or before). This could happen since commit
05ae571241, which can require skipping packets (so they immediately end
up in the backbuffer).
With the timeline code, a packet at the start or end of a segment can
refer to an invisible frame. So it doesn't extend the seek range, and
the timestamp should be clipped to the actual segment range.
Also restructure recompute_keyframe_target_pts() to be hopefully less
confusing.
Restores some behavior from before the demuxer cache changes, though
affects mostly just OSD display. The unknown queue state is reserved for
streams with missing or messed up timestamps.
Fully fixes behavior of the files mentioned in the previous commit. Will
probably lead to worse behavior if someone tries to fix real video and
cover art tracks, but that's a broken fringe case anyway.
This fixes .cue files with audio files that contain attached pictures to
some degree. demux_timeline.c just discarded packets with unset index,
so the picture was never fed to the decoder.
Directory-opening never worked on Windows because MSVCRT's open()
doesn't open directories and its fstat() doesn't recognise directory
handles. These are just MSVCRT restrictions, and the Windows API itself
has no problem with opening directories as file objects, so reimplement
mpv's mp_open and mp_stat to use the Windows API directly. This should
fix directory playback.
This also populates the st_dev and st_ino fields of struct stat, so
filesystem loop checking in demux_playlist.c should now work on Windows.
Fixes#4711
The new_segment field was used to track the decoder data flow handler of
timeline boundaries, which are used for ordered chapters etc. (anything
that sets demuxer_desc.load_timeline). This broke seeking with the
demuxer cache enabled. The demuxer is expected to set the new_segment
field after every seek or segment boundary switch, so the cached packets
basically contained incorrect values for this, and the decoders were not
initialized correctly.
Fix this by getting rid of the flag completely. Let the decoders instead
compare the segment information by content, which is hopefully enough.
(In theory, two segments with same information could perhaps appear in
broken-ish corner cases, or in an attempt to simulate looping, and such.
I preferred the simple solution over others, such as generating unique
and stable segment IDs.)
We still add a "segmented" field to make it explicit whether segments
are used, instead of doing something silly like testing arbitrary other
segment fields for validity.
Cached seeking with timeline stuff is still slightly broken even with
this commit: the seek logic is not aware of the overlap that segments
can have, and the timestamp clamping that needs to be performed in
theory to account for the fact that a packet might contain a frame that
is always clipped off by segment handling. This can be fixed later.
Needed for a failed thing, leaving it anyway because it causes no harm
and might be less awkward if struct virtual_stream is possibly extended
anyway in the future.
Although seeking past the cached range will trigger a low level seek, a
seek into the region between cache end and last video key frame would
simply seek to the video key frame. This meant that you could get
"stuck" at the end of the file instead of terminating playback when
trying to seek past the end.
One change is that we fix this by _actually_ allowing SEEK_FORWARD to
seek past the last video keyframe in find_seek_target().
In that case, or otherwise seeking to cache buffer end, it could happen
that we set ds->reader_head=NULL if the seek target is after the current
packet. We allow this, because the end of the cached region is defined
by the existence of "any" packet, not necessarily a key frame. Seeking
there still makes sense, because we know that there is going to be more
packets (or EOF) that satisfy the seek target.
The problem is that just resuming demuxing with reader_head==NULL will
simply return any packets that come its way, even non-keyframe ones.
Some decoders will produce ugly soup in this case. (In practice, this
was not a problem, because seeking at the end of the cached region was
rare before this commit, and also some decoders like h264 will skip
broken frames by default anyway.)
So the other change of this commit is to enable key frame skipping.
As a nasty implementation detail, we use a separate flag, instead of
setting reader_head to the first key frame encounted (reader_head being
NULL can happen after a normal seek or on playback start, and then we
want to mirror the underlying demuxer behavior, for better or worse).
This change is relatively untested, so you get to keep the pieces for
yourself.
Seems like most code dealing with this was for setting it in redundant
cases. Now SEEK_BACKWARD is redundant, and SEEK_FORWARD is the odd one
out.
Also fix that SEEK_FORWARD was not correctly unset in try_seek_cache().
In demux_mkv_seek(), make the arbitrary decision that a video stream is
not required for the subtitle prefetch logic to be active. We might want
subtitles with long duration even with audio only playback, or if the
file is used as external subtitle.
If a packet uses segmentation, the codec field must be set. Copying the
codec field was forgotten as an oversight, which is why this just
crashes. This showed up only now, because demux_copy_packet() was not
used before in the main demux path until recently.
Fixes#5027.
This improves upon the previous commit, and partially rewrites it (and
other code). It does:
- disable the seeking within cache by default, and add an option to
control it
- mess with the buffer estimation reporting code, which will most likely
lead to funny regressions even if the new features are not enabled
- add a back buffer to the packet cache
- enhance the seek code so you can seek into the back buffer
- unnecessarily change a bunch of other stuff for no reason
- fuck up everything and vomit ponies and rainbows
This should actually be pretty usable. One thing we should add are some
properties to report the proper buffer state. Then the OSC could show a
nice buffer range. Also configuration of the buffers could be made
simpler. Once this has been tested enough, it can be enabled by default,
and might replace the stream cache's byte ringbuffer.
In addition it may or may not be possible to keep other buffer ranges
when seeking outside of the current range, but that would be much more
complex.
More the ignore_eof field to the internal demux_stream struct. This is
relatively messy, because the internal struct exists only once the
stream is created, and after that setting the ignore_eof flag is a race
condition. We could bother with adding demux_add_sh_stream() parameters
for this, but let's not. So in theory a tiny race condition is
introduced, which can never be triggered since all demux API functions
are called by the playback thread only anyway.
Fix that ts_offset is accessed without log (this was introduced much
earlier by myself).
Introduce an alternative way of avoiding the annoying EOF reached
messages by not resetting the EOF flags for CC streams when a CC packet
is added. This makes the second commit in the PR which added the
original fix unnecessary.
As another cosmetic change merge the check in cached_demux_control()
into a single if().
In the future, the CC pseudo-stream should probably be replaced with an
entire pseudo-demuxer or such, which would avoid some of the messiness
(or maybe not, we don't know yet).
iive agreed to relicense things that are still in mpv to LGPLv2.1. So
change the licenses of the affected files, and rename the configure
switch for LGPL mode to --enable-preliminary-lgpl2.
(The "preliminary" part will probably be removed from the configure
switch soon as well.)
Also player/main.c hasn't had GPL parts since a few commits ago.
See "Copyright" file for caveats.
This changes the remaining "almost LGPL" files to LGPL, because we think
that the conditions the author set for these was finally fulfilled.
This adds handling of spherical video metadata: retrieving it from
demux_lavf and demux_mkv, passing it through filters, and adjusting it
with vf_format. This does not include support for rendering this type of
video.
We don't expect we need/want to support the other projection types like
cube maps, so we don't include that for now. They can be added later as
needed.
Also raise the maximum sizes of stringified image params, since they
can get really long.
If --demuxer-mkv-probe-start-time=no is used, and a seek is triggered on
start, then cluster_start will be 0, and the packet reading code will
print an error message about not finding valid data. This fixes itself
since it invokes the resync code, but it's still pretty ugly. Avoid this
by always initializing cluster_start.
