This allows users to control whether full dialogue subtitles are displayed
with an audio track already in their preferred subtitle language.
Additionally, this improves handling for the forced flag, automatically
selecting between forced and unforced subtitle streams based on the user's
settings and the selected audio.
Seems like this is requested all the time.
It seems libass allows out of range values, but does allows the subtitle
to go out of the screen at the bottom (only when moving it to the top
it's "clamped"). Too bad, don't do that then. The bitmap sub rendering
code on the other hand is under our control, and will not move a
subtitle out of the screen.
Fixes: #7986
Options like --sub-ass-force-style and others could not be changed at
runtime (the changes didn't take any effect). Fix this by using the
brutal approach, and completely reinit the subtitle state when this
happens. Maybe a bit clunky, but for now I'd rather not put more effort
into this.
Fixes: #7689
libass recently switched the default from 1 to 0 for compatibility
with ASS scripts that rely on the historical/VSFilter default of 0.
libass does attempt to detect and avoid breaking scripts that rely
on the historic libass-only default of 1, but it doesn't cover tracks
created directly through the API, so set the header explicitly.
Fixes https://github.com/mpv-player/mpv/issues/7900.
Remove the vaguely defined plane_bits and component_bits fields from
struct mp_imgfmt_desc. Add weird replacements for existing uses. Remove
the bytes[] field, replace uses with bpp[].
Fix some potential alignment issues in existing code. As a compromise,
split mp_image_pixel_ptr() into 2 functions, because I think it's a bad
idea to implicitly round, but for some callers being slightly less
strict is convenient.
This shouldn't really change anything. In fact, it's a 100% useless
change. I'm just cleaning up what I started almost 8 years ago (see
commit 00653a3eb0). With this I've decided to keep mp_imgfmt_desc,
just removing the weird parts, and keeping the saner parts.
See manpage additions. This was requested, sort of. Although what has
been requested might be something completely different. So this is
speculative.
This also changes sub_get_text() to return an allocated copy, because
the buffer shit was too damn messy.
Whatever it's worth. Instead of doing a pretty stupid conversion to
float, just blend it directly. This works for most RGB formats that are
8 bits per component or below (the latter because we expand packed
fringe RGB formats for simplicity). For higher bit depth RGB this would
need extra code.
This checked params->color.space for being RGB. If the colorspace is
unset, this did dumb things because even if the imgfmt was a RGB one,
the colorspace was not set to RGB. This actually also happened to the
tests.
(Short-cutting RGB like this is actually wrong, since RGB could still
have strange gamma or primaries, which would warrant a full conversion.
So you'd need to check for these other parameters as well. To be fixed
later.)
Maybe this is useful for some of the lesser VOs. It's preferable over
bad ad-hoc solutions based on the more complex sub_bitmap data
structures (as observed e.g. in vo_vaapi.c), and does not use that much
more code since draw_bmp already created such an overlay internally.
But I still wanted something that avoids having to upload/render a full
screen-sized overlay if for example there's only a tiny subtitle line on
the bottom of the screen. So the new API can return a list of modified
pixels (for upload) and non-transparent pixels (for display). The way
these pixel rectangles are computed is a bit dumb and returns dumb
results, but it should be usable, and the implementation can change.
They are sort of confusing, and they hide the fact that they have an
alpha component. Using the actual formats directly is no problem, sicne
these were useful only for big endian systems, something we can't test
anyway.
draw_bmp.c is the software blender for subtitles and OSD. It's used by
encoding mode (burning subtitles), and some VOs, like vo_drm, vo_x11,
vo_xv, and possibly more.
This changes the algorithm from upsampling the video to 4:4:4 and then
blending to downsampling the OSD and then blending directly to video.
This has far-reaching consequences for its internals, and results in an
effective rewrite.
Since I wanted to avoid un-premultiplying, all blending is done with
premultiplied alpha. That's actually the sane thing to do. The old code
just didn't do it, because it's very weird in YUV fixed point.
Essentially, you'd have to compensate for the chroma centering constant
by subtracting src_alpha/255*128. This seemed so hairy (especially with
correct rounding and high bit depths involved) that I went for using
float.
I think it turned out mostly OK, although it's more complex and less
maintainable than before. reinit() is certainly a bit too long. While it
should be possible to optimize the RGB path more (for example by
blending directly instead of doing the stupid float conversion), this is
probably slower. vo_xv users probably lose in this, because it takes the
slowest path (due to subsampling requirements and using YUV).
Why this rewrite? Nobody knows. I simply forgot the reason. But you'll
have it anyway. Whether or not this would have required a full rewrite,
at least it supports target alpha now (you can for example hard sub
transparent PNGs, if you ever wanted to use mpv for this).
Remove the check in vf_sub. The new draw_bmp.c is not as reliant on
libswscale anymore (mostly uses repack.c now), and osd.c shows an
error message on missing support instead now.
Formats with chroma subsampling of 4 are not supported, because FFmpeg
doesn't provide pixfmt definitions for alpha variants. We could provide
those ourselves (relatively trivial), but why bother.
The OSD is passed to VOs via struct sub_bitmaps, which has a change_id
field. This field is incremented whenever there is a (potential) change
to the other struct contents. If not, the VO can rely on it not having
changed. This must include for example sub_bitmap.x and sub_bitmap.dw.
If these two fields (and y equivalents) change, change_id must change,
even if the subtitle bitmap data might still be the same.
sd_lavc.c stopped respecting this at some unknown point. It could
sometimes cause problems, though usually only with bad and old VOs which
somehow relied on this more than vo_gpu. (I've actually encountered this
before with sd_lavc subtitle scaling, as indicated by a nasty comment,
though probably didn't track this down, since said old VOs can die in a
fire.)
Fix this by maintaining the change_id explicitly. Unfortunately adds
even more code. Instead of comparing the result we could track property
changes, but I think this is better. The number of parts is always very
low with this subtitle decoder, so there's no actual performance issue
to worry about.
This could be triggered by scaling changes (video-zoom etc.), but
probably also changing bitmap subtitle position or scaling.
Should be somewhat helpful. (All VOs are full of code trying to
compensate for this, more or less, and this will allow simplifying
some code later. Maybe.)
The screen size is mostly for robustness checks.
Making OSD/subtitle bitmaps refcounted was planend a longer time ago,
e.g. the sub_bitmaps.packed field (which refcounts the subtitle bitmap
data) was added in 2016. But nothing benefited much from it, because
struct sub_bitmaps was usually stack allocated, and there was this weird
callback stuff through osd_draw().
Make it possible to get actually refcounted subtitle bitmaps on the OSD
API level. For this, we just copy all subtitle data other than the
bitmaps with sub_bitmaps_copy(). At first, I had planned some fancy
refcount shit, but when that was a big mess and hard to debug and just
boiled to emulating malloc(), I made it a full allocation+copy. This
affects mostly the parts array. With crazy ASS subtitles, this parts
array can get pretty big (thousands of elements or more), in which case
the extra alloc/copy could become performance relevant. But then again
this is just pure bullshit, and I see no need to care. In practice, this
extra work most likely gets drowned out by libass murdering a single
core (while mpv is waiting for it) anyway. So fuck it.
I just wanted this so draw_bmp.c requires only a single call to render
everything. VOs also can benefit from this, because the weird callback
shit isn't necessary anymore (simpler code), but I haven't done anything
about it yet. In general I'd hope this will work towards simplifying the
OSD layer, which is prerequisite for making actual further improvements.
I haven't tested some cases such as the "overlay-add" command. Maybe it
crashes now? Who knows, who cares.
In addition, it might be worthwhile to reduce the code duplication
between all the things that output subtitle bitmaps (with repacking,
image allocation, etc.), but that's orthogonal.
UB sanitizer complains that aval<<24 (if >=128) cannot be represented as
int. Indeed, we would shift a bit into the sign of an int, which is
probably UB or implementation defined (I can't even remember, but the
stupidity of it burns). So technically, ubsan might be right.
Change aval to uint32_t, which I don't think has a chance of getting
promoted to int. Change the other *val to uint32_t too for cosmetic
symmetry.
So we have to obscure the intention of the code (*val can take only 8
bits) out of language stupidity. How nice. (What a shitty language.)
A mkv sample file was provided to me, which contained a moving PGS
subtitle track, with the same track rendered into the video as
reference. The subtitle track appeared to stutter (while the video one
was smooth). It turns out this was a timestamp rounding issue in mpv.
The subtitle timestamps in the file match the video ones exactly.
They're the same within the mpv demuxer too. Unfortunately, the
conversion from and to libavcodec timestamps is lossy, because mpv uses
a non-integer timebase, while libavcodec supports integers only. See
mp_pts_to_av() and mp_pts_from_av(). The recovered timestamp is almost
the same, but is off by a very minor part. As a result, the timestamps
won't compare equal, and if that happens, display of the subtitle frame
is skipped. Subtitle timestamps don't go through this conversion
because... libavcodec is special? The libavcodec subtitle API is
special.
Fix this by giving it a microsecond of slack. This is basically as if we
used an internal microseconds integer timebase, but only for the purpose
of image subtitle display.
The same could happen to sd_ass, except in practice it doesn't. ASS
subtitles (well, .ass files) inherently use a timebase incompatible to
video, so to ensure frame exactness, ASS timestamps are usually set to
slightly before the video frame's.
Discussion of better solutions:
One could rewrite mpv not to use float timestamps. You'd probably pick
some integer timebase instead (like microseconds), which would avoid the
libavcodec interop issue. At the very least this would be a lot of work.
It would be interesting to know whether the rounding in ther mpv<->lavc
timestamp conversion could be fixed to round-trip in this case. The
conversion tries to avoid problems by using the source timebase (e.g.
milliseconds with mkv). But in general some rounding is unavoidable,
because something between decoder and lowest demuxer layer could
transform the timestamps.
One could extend libavcodec to attach arbitrary information to avpacket
and return it in the resulting avframe. To some degree, such a mechanism
already exists (side data). But there are certain problems that make
this unfeasible and broken.
One could pass through exact mpv float timestamps by reinterpret-casting
them to int64_t, the FFmpeg timestamp type. Actually mpv used to do
this. But there were problems, such as FFmpeg (or things used by FFmpeg)
wanting to interpret the timestamps. Awful shit that make mpv change to
the current approach.
There's probably more but I'm getting bored. With some luck I wasted
precious seconds of your life with my nonsense.
Add an infrastructure for collecting performance-related data, use it in
some places. Add rendering of them to stats.lua.
There were two main goals: minimal impact on the normal code and normal
playback. So all these stats_* function calls either happen only during
initialization, or return immediately if no stats collection is going
on. That's why it does this lazily adding of stats entries etc. (a first
iteration made each stats entry an API thing, instead of just a single
stats_ctx, but I thought that was getting too intrusive in the "normal"
code, even if everything gets worse inside of stats.c).
You could get most of this information from various profilers (including
the extremely primitive --dump-stats thing in mpv), but this makes it
easier to see the most important information at once (at least in
theory), partially because we know best about the context of various
things.
Not very happy with this. It's all pretty primitive and dumb. At this
point I just wanted to get over with it, without necessarily having to
revisit it later, but with having my stupid statistics.
Somehow the code feels terrible. There are a lot of meh decisions in
there that could be better or worse (but mostly could be better), and it
just sucks but it's also trivial and uninteresting and does the job. I
guess I hate programming. It's so tedious and the result is always shit.
Anyway, enjoy.
Using mpv without libass isn't really supported, since it's not only
used to display ASS subtitles, but all text subtitles, and even OSD.
At least 1 user complained that the player printed a warning if built
without libass. Avoid trying to create the impression that using this
software without libass is in any way supported or desirable, and make
it fully mandatory.
(As far as making dependencies optional goes, I'd rather make ffmpeg
optional, which is an oversized and bloated library, rather than
something tiny like libass.)
Change all OPT_* macros such that they don't define the entire m_option
initializer, and instead expand only to a part of it, which sets certain
fields. This requires changing almost every option declaration, because
they all use these macros. A declaration now always starts with
{"name", ...
followed by designated initializers only (possibly wrapped in macros).
The OPT_* macros now initialize the .offset and .type fields only,
sometimes also .priv and others.
I think this change makes the option macros less tricky. The old code
had to stuff everything into macro arguments (and attempted to allow
setting arbitrary fields by letting the user pass designated
initializers in the vararg parts). Some of this was made messy due to
C99 and C11 not allowing 0-sized varargs with ',' removal. It's also
possible that this change is pointless, other than cosmetic preferences.
Not too happy about some things. For example, the OPT_CHOICE()
indentation I applied looks a bit ugly.
Much of this change was done with regex search&replace, but some places
required manual editing. In particular, code in "obscure" areas (which I
didn't include in compilation) might be broken now.
In wayland_common.c the author of some option declarations confused the
flags parameter with the default value (though the default value was
also properly set below). I fixed this with this change.
This is more or less a minimal hack to make _some_ text measurement
functionality available to scripts. Since libass does not support such a
thing, this simply uses the bounding box of the rendered text.
This is far from ideal. Problems include:
- using a bitmap bounding box
- additional memory waste and/or flushing caches
- dependency on window size
- odd small deviations with different window sizes (run osd-test.lua and
resize the window after each timer update; the bounding boxes aren't
adjusted in an overly useful way)
- inability to query the size _after_ actual rendering
But I guess it's a start. Since I'm aware that it's crap, add a threat
to the manpage that this may be changed/removed again. For now, I'm
interested whether anyone will have use for it in its current form, as
it's an often requested feature.
Setting demux_set_stream_wakeup_cb() will make all sh_stream (i.e.
track) specific wakeups go to this callback. But the callback takes care
of only the sub_preload() case (where it tries to pre-load subtitles
from already parsed and memory-present subtitles in a blocking way).
The old code assumed that the normal demuxer wakeup callback is called.
This was disregarded when the newer code was added. (And actually, the
original plan was to make _all_ per-sh_stream wakeups go to specialized
callbacks to avoid wasted work. dec_sub really should set the callback
always, and propagate wakeups to the playloop code. But it's too far
into the night to write coherent code.)
I couldn't actually observe any manifestation of this bug. Normally, the
playloop wakes up for other reasons (such as driving audio and video
decoding), so the lost wakeups rarely matter.
A negative subtitle delay means that subtitles from the future should be
shown earlier. With muxed subtitles, subtitle packets are demuxed along
with audio and video packets. But since they are demuxed "lazily",
nothing guarantees that subtitle packets from the future are available
in time.
Typically, the user-observed effect is that subtitles do not appear at
all (or too late) with large negative --sub-delay values, but that using
--cache might fix this.
Make this behave better. Automatically extend read-ahead to as much as
needed by the subtitles. It seems it's the easiest to pass the subtitle
render timestamp to the demuxer in order to guarantee that everything is
read. This timestamp based approach might be fragile, so disable it if
no negative sub-delay is used.
As far as the player frontend part is concerned, this makes use of the
code path for external subtitles, which are not lazily demuxed, and may
already trigger waiting.
Fixes: #7484
Libav seems rather dead: no release for 2 years, no new git commits in
master for almost a year (with one exception ~6 months ago). From what I
can tell, some developers resigned themselves to the horrifying idea to
post patches to ffmpeg-devel instead, while the rest of the developers
went on to greener pastures.
Libav was a better project than FFmpeg. Unfortunately, FFmpeg won,
because it managed to keep the name and website. Libav was pushed more
and more into obscurity: while there was initially a big push for Libav,
FFmpeg just remained "in place" and visible for most people. FFmpeg was
slowly draining all manpower and energy from Libav. A big part of this
was that FFmpeg stole code from Libav (regular merges of the entire
Libav git tree), making it some sort of Frankenstein mirror of Libav,
think decaying zombie with additional legs ("features") nailed to it.
"Stealing" surely is the wrong word; I'm just aping the language that
some of the FFmpeg members used to use. All that is in the past now, I'm
probably the only person left who is annoyed by this, and with this
commit I'm putting this decade long problem finally to an end. I just
thought I'd express my annoyance about this fucking shitshow one last
time.
The most intrusive change in this commit is the resample filter, which
originally used libavresample. Since the FFmpeg developer refused to
enable libavresample by default for drama reasons, and the API was
slightly different, so the filter used some big preprocessor mess to
make it compatible to libswresample. All that falls away now. The
simplification to the build system is also significant.
Works as ad-filter. I had some more plans, for example replacing
matching text with different text, but for now it's dropping matches
only. There's a big warning in the manpage that I might change
semantics. For example, I might turn it into a primitive sed.
In a sane world, you'd probably write a simple script that processes
downloaded subtitles before giving them to mpv, and avoid all this
complexity. But we don't live in a sane world, and the sooner you learn
this, the happier you will be. (But I also want to run this on muxed
subtitles.)
This is pretty straightforward. We use POSIX regexes, which are readily
available without additional pain or dependencies. This also means it's
(apparently) not available on win32 (MinGW). The regex list is because I
hate big monolithic regexes, and this makes it slightly better.
Very superficially tested.
Until now, filter_sdh was simply a function that was called by sd_ass
directly (if enabled).
I want to add another filter, so it's time to turn this into a somewhat
more general subtitle filtering infrastructure.
I pondered whether to reuse the audio/video filtering stuff - but better
not. Also, since subtitles are horrible and tend to refuse proper
abstraction, it's still messed into sd_ass, instead of working on the
dec_sub.c level. Actually mpv used to have subtitle "filters" and even
made subtitle converters part of it, but it was fairly horrible, so
don't do that again.
In addition, make runtime changes possible. Since this was supposed to
be a quick hack, I just decided to put all subtitle filter options into
a separate option group (=> simpler change notification), to manually
push the change through the playloop (like it was sort of before for OSD
options), and to recreate the sub filter chain completely in every
change. Should be good enough.
One strangeness is that due to prefetching and such, most subtitle
packets (or those some time ahead) are actually done filtering when we
change, so the user still needs to manually seek to actually refresh
everything. And since subtitle data is usually cached in ASS_Track (for
other terrible but user-friendly reasons), we also must clear the
subtitle data, but of course only on seek, since otherwise all subtitles
would just disappear. What a fucking mess, but such is life. We could
trigger a "refresh seek" to make this more automatic, but I don't feel
like it currently.
This is slightly inefficient (lots of allocations and copying), but I
decided that it doesn't matter. Could matter slightly for crazy ASS
subtitles that render with thousands of events.
Not very well tested. Still seems to work, but I didn't have many test
cases.
This is for easier use with the "delay_open" feature added in the
previous commit. The "null" codec is reported if the codec is unknown
(because the stream was not opened yet at time the tracks were added).
The rest of the timeline mechanism will set the correct codec at
runtime. But this means every time a delay-loaded track is selected, it
wants to initialize a decoder for the "null" codec.
Accept a "null" decoder. But since FFmpeg has no such codec, and out of
my own laziness, just let it fall back to "common" codecs that need no
other initialization data.
The change, in an earlier commit, in format for ass to handle results
in a different number of fields to skip. Correct that so SDH filtering
works.
Should fix issue #7188
check_obj_resize() in sub/osd.c calls mp_client_broadcast_event(), which
calls notify_property_events(). This is pretty unexpected, because
check_obj_resize() may be called from the VO thread. While that's sort
of awful, it seems to be OK locking-wise. But it breaks an assumption in
notify_property_events() that the core doesn't need to be woken up,
which could possibly lead to a missed/delayed property update (although
rather unlikely).
Fix this by explicitly waking up the core when it's called from the OSD
code.
Lua scripting has an undocumented mp.set_osd_ass() function, which is
used by osc.lua and console.lua. Apparently, 3rd party scripts also use
this. It's probably time to make this a public API.
The Lua implementation just bypassed the libmpv API. To make it usable
by any type of client, turn it into a command, "osd-overlay".
There's already a "overlay-add". Ignore it (although the manpage admits
guiltiness). I don't really want to deal with that old command. Its main
problem is that it uses global IDs, while I'd like to avoid that scripts
mess with each others overlays (whether that is accidentally or
intentionally). Maybe "overlay-add" can eventually be merged into
"osd-overlay", but I'm too lazy to do that now.
Scripting now uses the commands. There is a helper to manage OSD
overlays. The helper is very "thin"; I only want to force script authors
to use the ID allocation, which may help with putting multiple scripts
into a single .lua file without causing conflicts (basically, avoiding
singletons within a script's environment). The old set_osd_ass() is
emulated with the new API.
The JS scripting wrapper also provides a set_osd_ass() function, which
calls internal mpv API. Comment that part (to keep it compiling), but
I'm leaving it to @avih to finish the change.
I was recently informed that unicode has official symbols for
window controls, and I put together a change to use them, which
worked, as long as a suitable font was installed. However, it's
not that hard to get a normal system that lacks an appropriate
font, and libass wants to print warnings if the symbols aren't
in the default font, which will almost always be true.
So, I gave up and added the symbols to the custom osd font that
we already have. This ensures they are always available, and
that they are aligned consistently on all platforms.
I took the symbols from the `symbola` font, as this has a suitable
licence and the symbols look nice enough.
Symbola Licence:
Fonts are free for any use; they may be opened, edited,
modified, regenerated, packaged and redistributed.
Finally, as we now have access to an un-maximize symbol, I added
logic to use it when the window is maximized.
Do what we do best in multimedia: add conflicting hacks on top of other
hacks, that fix a single sample, and may break other ones.
In this case, it only happens if the file is most likely already broken
(subtitle bounding boxes go outside of the subtitle "canvas"), so it's
OK. The file still looks broken (and, in fact, the file is completely
fucking broken), but you can see the subtitles.
But in summary, this is not actually something I should have bothered
about.
I noticed that MPlayer shows the subtitles "correctly", but this is only
because they have a hack that extends subtitles with small resolution to
a larger hardcoded resolution. This hack was removed from mpv, because
it broke some completely legitimate files. As another really funny fact,
MPlayer's default video output (vdpau) appears to display this file
correctly, but only because it handles narrow aspect ratios (that extend
the height instead of the width) incorrectly. It extends the height, but
leaves the video with 1:1 aspect ratio at the top. It seems to repeat
the last video line. (-vo xv and -vo gl show it correctly, i.e.
stretched like mpv, by the way.) For some reason, the sample file at
hand is extended with black, so the subtitles are rendered into a black
area below the video, which is almost reasonable. So, MPlayer may
display this file "correctly", but in fact it only happens to do so
because of 1 hack that breaks legitimate files, and 1 bug. What the
fuck.
Fixes: #7218 (sort of)
Sometimes the atlas can get so large that it exceeds the maximum allowed
size for an mp_image. Since the atlas will never shrink naturally, this
breaks subtitles entirely until mpv is restarted. Reset the packer so
that subtitles can rendered properly once the atlas fits again.
This is a partial workaround for #6286.
libass had an API to configure this since 2013. mpv always used
ASS_FONTPROVIDER_AUTODETECT, because usually there's little reason to
use anything else. The intention of the now added option is to allow
users to disable use of system fonts.
I didn't consider it worth the trouble to add the coretext and
directwrite enum items from ASS_DefaultFontProvider. The "auto" choice
will have the same effect if they're available. Also, the part of the
code which defines the option does not necessarily have libass available
(it's still optional!), so defining all enum items as choices is icky. I
still added fontconfig, since that may be nice to emulate a nostalgic
2010 feeling of mpv freezing on fontconfig.
The option for OSD is even less useful. (But you get it for free, and
why pass up a chance to add yet another useless option?)
This is not quite what was requested in #6947, but as close as it gets.
First we shift the values up to the actual amount of bits in draw_ass,
so that they will be drawn correctly when using formats with more than
8 bpc. (draw_rgba is already correct w.r.t. RGB formats with 9 or more
bpc)
Then, in scale_sb_rgba, by setting the amount of bits per channel used
for planar RGB formats (formats are always planar at this point in
draw_bmp) to be the same as the source from 9 to 16 bpc (in effect all
the various GBRP formats) we manage to fit the special case that does
not require any conversion in chroma_up and chroma_down when handling
these formats (as long as the source itself is a planar format),
instead writing directly to the combined dst/src buffer. This in turn
works around a bug (incorrect colors) in libswscale when scaling
between GBRP formats with 9 or more bpc. Additionally this should be
more efficient, since we skip up- and down-conversion and temporary
buffers.
During initial testing with US closed captions, ARIB captions,
timed text in MP4 or the specific external SRT files I tested with
there were no hints that this flag would be needed for seeking to
work.
Unfortunately, that result seems to have been incorrect.
Fixes#6970
