The previous commit means subtitles were reinitialized on every seek
(even within a segment). This commit restores the old behavior.
To check whether the segment changed at all, we don't reset the current
start/end values. This assumes the decoder wrapper is always fed by a
stream which doesn't mix segment and non-segment packets, which is
currently always true.
The accepts_packet packet callback is supposed to deal with subtitle
decoders which have only a small queue of current subtitle events (i.e.
sd_lavc.c), in case feeding it too many packets would discard events
that are still needed.
Normally, the number of subtitles that need to be preserved is estimated
by the rendering pts (get_bitmaps() argument). Rendering lags behind
decoding, so normally the rendering pts is smaller than the next video
frame pts, and we simply discard all subtitle events until the rendering
pts.
This breaks down in some annoying corner cases. One of them is seeking
backwards: the VO will still try to render the old PTS during seeks,
which passes a high PTS to the subtitle renderer, which in turn would
discard more subtitles than it should. There is a similar issue with
forward seeks. Add hacks to deal with those issues.
There should be a better way to deal with the essentially unknown
"rendering position", which is made worse by screenshots or rendering
with vf_sub. At the very least, we could handle seeks better, and e.g.
either force the VO not to re-render subs after seeks (ugly), or
introduce seek sequence numbers to distinguish attempts to render
earlier subtitles when a seek is done.
The intention is to let mp_ass_packer_pack() produce different output
for the RGBA and LIBASS formats. VOs (or whatever generates the OSD)
currently do not signal a preferred format, and this mechanism just
exists to switch between RGBA and LIBASS formats correctly, preferring
LIBASS if the VO supports it.
In particular, this prevents subtitle packets from building up in the
subtitle queue if e.g. --vo=null is used. In this situation,
sub_get_bitmaps() is never called, and thus the segment never switched.
This also seems to help with flickering at segment switch boundaries (if
subs are supposed to be visible at the transition points).
In theory, this could trigger a switch too early, but the way VO and
subtitle renderer interact wrt. timing is a bit iffy anyway.
Instead of passing an explicit cache to the function, the res parameter
is used. Also, instead of replacing its contents, sub bitmaps are now
appended to it (all assuming the format doesn't actually change).
This is preparation for the following commits.
Subtitles can be preloaded, which means they're fully read and copied
into ASS_Track. This in turn is mainly for the sake of being able to do
subtitle seeking (when it comes down to it, subtitle seeking is the
cause for most trouble here).
Commit a714f8e92 broke preloaded subtitles which have events with
unknown duration, such as some MicroDVD samples. The event list gets
cleared on every seek, so the property of being preloaded obviously gets
lost.
Fix this by moving most of the preloading logic to dec_sub.c. If the
subtitle list gets cleared, they are not considered preloaded anymore,
and the logic for demuxed subtitles is used.
As another minor thing, preloadeding subtitles did neither disable the
demux stream, nor did it discard packets. Thus you could get queue
overflows in theory (harmless, but annoying). Fix this by explicitly
discarding packets in preloaded mode.
In summary, now the only difference between preloaded and normal
demuxing are:
1. a seek is issued, and all packets are read on start
2. during playback, discard the packets instead of feeding them to the
subtitle decoder
This is still petty annoying. It would be nice if maintaining the
subtitle index (and maybe a subtitle packet cache for instant subtitle
presentation when seeking back) could be maintained in the demuxer
instead. Half of all file formats with interleaved subtitles have
this anyway (mp4, mkv muxed with newer mkvmerge).
Commit 8d4a179c made subtitle decoders pick up fonts strictly from the
same source file (i.e. the same demuxer).
It breaks some fucked up use-case, and 2 people on this earth complained
about the change because of this. Add it back.
This copies all attached fonts on each subtitle init. I considered
converting attachments to use refcounting, but it'd probably be much
more complex.
Since it's slightly harder to get a list of active demuxers with
duplicate removed, the prev_demuxer variable serves as a hack to achieve
almost the same thing, except in weird corner cases. (In which fonts
could be added twice.)
This happens only if the new segment wasn't read yet.
This is not quite proper and a problem with dec_sub.c internals.
Ideally, it'd wait with rendering until a new enough segment has been
read. Normally, the new segment is available immediately, so the end
will be automatically clipped by switching to the right segment in the
exact moment it's supposed to become effective.
Usually shouldn't cause any problems, though.
This uses a different method to piece segments together. The old
approach basically changes to a new file (with a new start offset) any
time a segment ends. This meant waiting for audio/video end on segment
end, and then changing to the new segment all at once. It had a very
weird impact on the playback core, and some things (like truly gapless
segment transitions, or frame backstepping) just didn't work.
The new approach adds the demux_timeline pseudo-demuxer, which presents
an uniform packet stream from the many segments. This is pretty similar
to how ordered chapters are implemented everywhere else. It also reminds
of the FFmpeg concat pseudo-demuxer.
The "pure" version of this approach doesn't work though. Segments can
actually have different codec configurations (different extradata), and
subtitles are most likely broken too. (Subtitles have multiple corner
cases which break the pure stream-concatenation approach completely.)
To counter this, we do two things:
- Reinit the decoder with each segment. We go as far as allowing
concatenating files with completely different codecs for the sake
of EDL (which also uses the timeline infrastructure). A "lighter"
approach would try to make use of decoder mechanism to update e.g.
the extradata, but that seems fragile.
- Clip decoded data to segment boundaries. This is equivalent to
normal playback core mechanisms like hr-seek, but now the playback
core doesn't need to care about these things.
These two mechanisms are equivalent to what happened in the old
implementation, except they don't happen in the playback core anymore.
In other words, the playback core is completely relieved from timeline
implementation details. (Which honestly is exactly what I'm trying to
do here. I don't think ordered chapter behavior deserves improvement,
even if it's bad - but I want to get it out from the playback core.)
There is code duplication between audio and video decoder common code.
This is awful and could be shareable - but this will happen later.
Note that the audio path has some code to clip audio frames for the
purpose of codec preroll/gapless handling, but it's not shared as
sharing it would cause more pain than it would help.
This covers source files which were added in mplayer2 and mpv times
only, and where all code is covered by LGPL relicensing agreements.
There are probably more files to which this applies, but I'm being
conservative here.
A file named ao_sdl.c exists in MPlayer too, but the mpv one is a
complete rewrite, and was added some time after the original ao_sdl.c
was removed. The same applies to vo_sdl.c, for which the SDL2 API is
radically different in addition (MPlayer supports SDL 1.2 only).
common.c contains only code written by me. But common.h is a strange
case: although it originally was named mp_common.h and exists in MPlayer
too, by now it contains only definitions written by uau and me. The
exceptions are the CONTROL_ defines - thus not changing the license of
common.h yet.
codec_tags.c contained once large tables generated from MPlayer's
codecs.conf, but all of these tables were removed.
From demux_playlist.c I'm removing a code fragment from someone who was
not asked; this probably could be done later (see commit 15dccc37).
misc.c is a bit complicated to reason about (it was split off mplayer.c
and thus contains random functions out of this file), but actually all
functions have been added post-MPlayer. Except get_relative_time(),
which was written by uau, but looks similar to 3 different versions of
something similar in each of the Unix/win32/OSX timer source files. I'm
not sure what that means in regards to copyright, so I've just moved it
into another still-GPL source file for now.
screenshot.c once had some minor parts of MPlayer's vf_screenshot.c, but
they're all gone.
This is mainly a refactor. I'm hoping it will make some things easier
in the future due to cleanly separating codec metadata and stream
metadata.
Also, declare that the "codec" field can not be NULL anymore. demux.c
will set it to "" if it's NULL when added. This gets rid of a corner
case everything had to handle, but which rarely happened.
Slightly change how it is decided when a new packet should be read.
Switch to demux_read_packet_async(), and let the player "wait properly"
until required subtitle packets arrive, instead of blocking everything.
Move distinguishing the cases of passive and active reading into the
demuxer, where it belongs.
Just so I can remove a few lines from dec_sub.c.
This is slightly inelegant, as the whole subtitle file has to be read
into memory, converted at once in memory, and then provided to
libavformat in an awkward way by creating a memory stream instead of
using demuxer->stream. It also won't be possible to force the charset on
subtitles in binary container formats - but this wasn't exposed before,
and we just hope this won't be ever needed. (One motivation was fixing
broken files with non-UTF8 muxed.) It also won't be possible to change
the charset on the fly, but this was not exposed either.
Just simplify by removing parts not needed anymore. This includes
merging dec_sub allocation and initialization (since things making
initialization complicated were removed), or format support queries (it
simply tries to create a decoder, and if that fails, tries the next
one).
So that the video FPs is not required at initialization, and can be set
later.
(As for whether this MicroDVD crap is worth the trouble to handle it
"correctly": MicroDVD files are unfortunately still around, and in at
least one case using the video FPS seemed to help indeed.)
Keeping ASS_Renderers around for a potentially large number of subtitle
tracks could lead to excessive memory usage, especially since the libass
cache is broken (caches even unneeded data), and might consume up to
~500MB of memory for no reason.
This includes the case of switching ordered chapter boundaries. It will
now be recreated on each timeline part switch. This shouldn't be much of
a problem with modern libass. (Older libass versions use fontconfig for
memory fonts, and will be very slow to reinitialize memory fonts.)
Apparently, this was replaced by the SD_CTRL_SET_VIDEO_PARAMS set
dimensions. But I can't find out when this happened - possibly, these
fields were never used by sd_lavc.c, and only by the (long removed)
MPlayer dvdsub decoder.
It was stupid. The only thing that still effectively used it was
sd_lavc_conv - all other "filters" were the subtitle decoder/renderers
for text (sd_ass) and bitmap (sd_lavc) subtitles.
While having a subtitle filter chain was interesting (and actually
worked in almost the same way as the audio/video ones), I didn't
manage to use it in a meaningful way, and I couldn't e.g. factor
secondary features like fixing subtitle timing into filters.
Refactor the shit and drop unneeded things as it goes.
This affects non-ASS text subtitles (those which go through libavcodec's
subtitle converter), which are muxed with video/audio. (Typically srt
subs in mkv.)
The problem is that seeking in the file can send a subtitle packet to
the decoder multiple times. These packets are interlaved with video,
and thus can't be all read when opening the file. Rather, subtitle
packets can essentially be randomly skipped or repeated (by seeking).
Until recently, this was solved by scanning the libass event list for
duplicates. Then our builtin srt-to-ass converter was removed, and
the problem was handled by fully clearing the subtitle list on each
seek.
This resulted in sub-seek not working properly for this type of file.
Since the subtitle list was cleared on seek, it was not possible to
do e.g. sub-seeks to subtitles before the current playback position.
Fix this by not clearing the list, and intead explicitly rejecting
duplicate packets. We use the packet file position was unique ID for
subtitles; this is confirmed working for most file formats (although
it is slightly risky - new demuxers may not necessarily set the file
position to something unique, or at all).
The list of seen packets is sorted, and the lookup uses binary search.
This is to avoid quadratic complexity when subtitles are added in
bulks, such as when opening a text subtitle file.
In some places, the code has to be adjusted to pass through the packet
file position correctly.
With the FFmpeg subtitle decoder used for _all_ non-ASS text subtitle
format, this code is simply unused now.
Ironically, the FFmpeg subtitle decoder does not handle things correctly
in a bunch of cases. Should it turn out they actually matter, they will
have to hack back.
The extend_event one is a candidate, although even though there were
allegedly files which need it, I couldn't get samples from the user who
originally reported such files. As such, extend_event was only confirmed
to handle trailing events with no (endless) duration like with MicroDVD
and LRC, but FFmpeg "fudges" these anyway, so no special handling is
needed.
This code also had logic to handle seeking with muxed srt subtitles,
which made the sub-seek command work. But this has been broken before
this commit already. Currently, seeking with muxed srt subs will clear
all subtitles, as the broken FFmpeg ASS format output by the libavcodec
subtitle converters does not check for duplicates. Since the subtitles
are all cleared, ass_step_sub() can not work properly and sub-seek can
not seek to already seen subtitles.
Slightly simpler, and removes the need to pre-read all subtitle packets.
This still does the subtitle charset conversion on the packet level
(instead converting when parsing the file), so in theory this still
could provide a way to change the charset at runtime. But maybe even
this should be removed, as FFmpeg is somewhat likely to get its own
charset detection and conversion mechanism in the future. (Would have
to keep the subtitle file in memory to allow changing the charset on
the fly, I guess.)
The FFmpeg subtitle converter does the same. There used to be some
deficiencies in FFmpeg's code, but it seems at least some of them have
been fixed. There also used to be the timestamp issue (see previous
commit messages), but this doesn't matter anymore. So no reason to
keep this code - get rid of it.
This can be dropped for the same reasons as in the previous commits. It
removes MicroDVD conversion support on Libav, although MicroDVD files
couldn't be read in the first place ever since demux_subreader.c was
removed.
This restored timestamps when demuxing srt subtitles in Libav, which
was important for avoiding slightly overlapping subtitles. Since the
way this works was changed, there is no real reason to maintain proper
timestamps anymore on this level - this can be dropped without issues.
The awkward "preprocess" step of putting the subtitles through single
filters before doing something else was made unnecessary by the recent
changes.
(Fun fact: I originally planned to move these extra things, like fixing
subtitle gaps/overlaps, to filters - but this would suffer from various
complications, and moving them to the renderers seems much simpler.)
I feel like it's better there. Note that there is no reduced
functionality, as bitmaps subs (i.e. not handled by sd_ass.c) were never
fully read on init, and thus never went through sub_read_all_packets().
On the other hand, this might lead to confusion, as --sub-fps etc. will
now also affect muxed subtitles (which makes not much sense).
Instead of messing with the subtitle packet timestamps, do it on output.
We work on the libass event list. If there is an unwanted gap or
overlap, we render the timestamp at another position where there is no
gap or overlap.
This is somewhat more robust, and even works with demuxed subs (to some
degree - depends whether the subs are prefected soon enough).
It's active even for native ASS subs. I wonder if this is a problem with
extended type setting. If it is, the heuristic that tries to avoid
interrupting such cases has to be improved.
While it probably would be ideal to do this after the subtitle decoder,
certain aspects are at least currently handled better in this place.
Until now, feeding packets to the decoder in advance was done for text
subtitles only. This was possible because libass buffers all subtitle
data anyway (in ASS_Track). sd_lavc, responsible for bitmap subs, does
not do this. But it can buffer a small number of subtitle frames ahead.
Enable this.
Repurpose the sub_accept_packets_in_advance(). Instead of "can take all
packets" it means "can take 1 packet" now. (The old meaning is still
needed locally in dec_sub.c; keep it there.) It asks the decoder whether
there is place for at least 1 subtitle packet. sd_lavc implements it and
returns true if its internal fixed-size subtitle queue still has a free
slot. (The implementation of this in dec_sub.c isn't entirely clean.
For one, decode_chain() ignores this mechanism, so it's implied that
bitmap subtitles do not use the subtitle filter chain in any advanced
way.)
Also fix 2 bugs in the sd_lavc queue handling. Subtitles must be checked
in reverse, because the first entry will often have endpts==NOPTS, which
would always match. alloc_sub() must cycle the queue buffer, because it
reuses memory allocations (like sub.imgs) by design.
uchardet is written in C++, and thus doesn't appreciate the value of
using static strings, and internally stores the guessed charset as
allocated std::string. Add a minimal hack to deal with this. (I don't
appreciate that the code is potentially harder to understand by
returning either a static or allocated string, but I do appreciate for
not having to litter the existing code with strdups.)
Each subtitle track gets its own decoder instance (sd_ass). But they use
a shared ASS_Renderer. This is done mainly because of fontconfig.
Initializing fontconfig is very slow when using it with memory fonts, so
there's a practical need to cache this memory font state, which is done
by not creating separate ASS_Renderers. This is very dirty and very
evil, but we probably can't get rid of it any time soon.
The shared ASS_Renderer was not properly synchronized. While the program
logic guarantees that only one sd_ass instance is visible at a time,
there are other interactions that require synchronization. In
particular, I suspect concurrent execution of mp_ass_configure_fonts()
and sd_ass.get_bitmaps cause issues in a newer libass development
branch.
So here's a shitty hack that hopefully fixes things, hopefully only
until libass becomes less dependent on fontconfig.
MPlayer traditionally had completely separate sh_ structs for
audio/video/subs, without a good way to share fields. This meant that
fields shared across all these headers had to be duplicated. This commit
deduplicates essentially the last remaining duplicated fields.
Prefer the builtin one again. libavcodec still uses the ASS packet
format that uses inline timestamps, so the packet timestamps are
ignored. This again leads to additional rounding of timestamps, because
the ASS storage format only has 10ms resolution (instead of 1ms
resolution like libass). This again can lead to unintentional overlaps
when converting subtitles. The internal MicroDVD converter avoids this,
because it always uses packet timestamps.
This is a deadlock caused by a lock order issue: sub/osd.c locks the OSD
first, then the subtitle decoder lock. player/sub.c does the reverse.
Fix this by discussing away the requirement for locking (see below),
which allows us to drop the broken sub lock. sub_get_text() still
acquires and releases the sub decoder lock, but it's not held at the
same time as the OSD lock anymore, so it should be fine.
Originally, the sub lock was acquired because sub_get_text() returns a
pointer to a mutable string. We simply declare that it's ok to call it
unlocked, as long as only 1 thread accesses it, which works out fine in
this case.
We certainly don't want to maintain and improve the internal converter,
but we still need the internal one for Libav. (In the Libav case,
demux_subreader.c will be used to read the MicroDVD file.)
wm4
Aman Gupta
Marcin Kurczewski