This fixes audio encoding crashing under ASan.
When extended_data != data, FFmpeg copies more pointers from
extended_data (= the number of channels) than there really
are for non-planar formats (= exactly 1), but that's not our fault.
Regardless, this commit makes it work in all common cases.
AVFrame doesn't have public code for pool allocation, so mpv does it
manually. AVFrame allocation is very tricky, so we added a bug.
This crashed with libopus encoding, but not some other audio codecs,
because the libopus libavcodec wrapper accesses AVFrame.data. Most code
tries to avoid accessing AVFrame.data and uses AVFrame.extended_data,
because using the former would subtly corrupt memory on more than 8
channels. The fact that this problem manifested only now shows that most
AVFrame consuming FFmpeg code indeed uses extended_data for audio.
Allow mp_aframe_clip_timestamps() to discard a spdif frame if it's
entirely out of the timestamp range. Just a minor thing that might make
handling these dumb formats easier.
Libav seems rather dead: no release for 2 years, no new git commits in
master for almost a year (with one exception ~6 months ago). From what I
can tell, some developers resigned themselves to the horrifying idea to
post patches to ffmpeg-devel instead, while the rest of the developers
went on to greener pastures.
Libav was a better project than FFmpeg. Unfortunately, FFmpeg won,
because it managed to keep the name and website. Libav was pushed more
and more into obscurity: while there was initially a big push for Libav,
FFmpeg just remained "in place" and visible for most people. FFmpeg was
slowly draining all manpower and energy from Libav. A big part of this
was that FFmpeg stole code from Libav (regular merges of the entire
Libav git tree), making it some sort of Frankenstein mirror of Libav,
think decaying zombie with additional legs ("features") nailed to it.
"Stealing" surely is the wrong word; I'm just aping the language that
some of the FFmpeg members used to use. All that is in the past now, I'm
probably the only person left who is annoyed by this, and with this
commit I'm putting this decade long problem finally to an end. I just
thought I'd express my annoyance about this fucking shitshow one last
time.
The most intrusive change in this commit is the resample filter, which
originally used libavresample. Since the FFmpeg developer refused to
enable libavresample by default for drama reasons, and the API was
slightly different, so the filter used some big preprocessor mess to
make it compatible to libswresample. All that falls away now. The
simplification to the build system is also significant.
It's hard to see what FFmpeg does or what its API requires. It looks
like the alignment in our own allocation code might be slightly too
lenient, but who knows. Even if this is not needed, upping the alignment
only wastes memory and doesn't do anything bad.
(Note that the only reason why we have our own code is because FFmpeg
doesn't even provide it as API. API users are forced to recreate this,
even if they have no need for custom allocation!)
The "amultiply" filter crashes in AVX mode on unaligned access if an
audio pointer is unaligned (on 32 or 64 bytes I assume).
A requirement that audio data needs to be aligned isn't documented
anywhere. In our case, the data is still sample- and channel-aligned,
which is completely sane. Sure, you can imagine optimizations which make
some algorithms even faster by requiring higher alignment. But, and this
is a big but, you shouldn't crash api users because you just invented a
new undocumented requirement. And even more importantly, your
user-crashing optimization won't matter because it's just a trivial
algorithm working on audio. You don't need to pretend to be an
optimization devil, and nobody will give you a prize for this. But no,
lets random make API users crash (and then probably blame them for it!)
for something that wouldn't matter at all.
Not to mention that they do "document" some requirements on _video_
data, yet their vf_crop probably can still produce unaligned video
pointers. Oh how hilarious that the same documentation also talks about
libswscale alignment requirements. (This is weird because libswscale is
just one of many, many things which consume video data. Also did you
know that zimg, written in C++ and using intrinsics, i.e. the antithesis
to FFmpeg development, is much faster than libswscale, easier to use,
and produces more correct results, even if you ignore that libswscale
flat out doesn't support some very important features?)
Fucking tired of this bullshit. Can't wait until someone comes up with a
better framework than this... (well let's not write this out).
Fix this by copying instead of adjusting the start pointer when skipping
samples. This makes general operations slower just to fix interoperating
with a single filter. Thank you for your "optimization", FFmpeg. Go die
in a fire.
Didn't check whether this is correct. It probably is? If the frame needs
to be copied (due to COW), and memory allocation fails, it just silently
(or audibly lol) doesn't skip samples, because a never-fail function can
suddenly fail. Well, who cares.
Fixes: #7141
See manpage additions. This is a huge hack. You can bet there are shit
tons of bugs. It's literally forcing square pegs into round holes.
Hopefully, the manpage wall of text makes it clear enough that the whole
shit can easily crash and burn. (Although it shouldn't literally crash.
That would be a bug. It possibly _could_ start a fire by entering some
sort of endless loop, not a literal one, just something where it tries
to do work without making progress.)
(Some obvious bugs I simply ignored for this initial version, but
there's a number of potential bugs I can't even imagine. Normal playback
should remain completely unaffected, though.)
How this works is also described in the manpage. Basically, we demux in
reverse, then we decode in reverse, then we render in reverse.
The decoding part is the simplest: just reorder the decoder output. This
weirdly integrates with the timeline/ordered chapter code, which also
has special requirements on feeding the packets to the decoder in a
non-straightforward way (it doesn't conflict, although a bugmessmass
breaks correct slicing of segments, so EDL/ordered chapter playback is
broken in backward direction).
Backward demuxing is pretty involved. In theory, it could be much
easier: simply iterating the usual demuxer output backward. But this
just doesn't fit into our code, so there's a cthulhu nightmare of shit.
To be specific, each stream (audio, video) is reversed separately. At
least this means we can do backward playback within cached content (for
example, you could play backwards in a live stream; on that note, it
disables prefetching, which would lead to losing new live video, but
this could be avoided).
The fuckmess also meant that I didn't bother trying to support
subtitles. Subtitles are a problem because they're "sparse" streams.
They need to be "passively" demuxed: you don't try to read a subtitle
packet, you demux audio and video, and then look whether there was a
subtitle packet. This means to get subtitles for a time range, you need
to know that you demuxed video and audio over this range, which becomes
pretty messy when you demux audio and video backwards separately.
Backward display is the most weird (and potentially buggy) part. To
avoid that we need to touch a LOT of timing code, we negate all
timestamps. The basic idea is that due to the navigation, all
comparisons and subtractions of timestamps keep working, and you don't
need to touch every single of them to "reverse" them.
E.g.:
bool before = pts_a < pts_b;
would need to be:
bool before = forward
? pts_a < pts_b
: pts_a > pts_b;
or:
bool before = pts_a * dir < pts_b * dir;
or if you, as it's implemented now, just do this after decoding:
pts_a *= dir;
pts_b *= dir;
and then in the normal timing/renderer code:
bool before = pts_a < pts_b;
Consequently, we don't need many changes in the latter code. But some
assumptions inhererently true for forward playback may have been broken
anyway. What is mainly needed is fixing places where values are passed
between positive and negative "domains". For example, seeking and
timestamp user display always uses positive timestamps. The main mess is
that it's not obvious which domain a given variable should or does use.
Well, in my tests with a single file, it suddenly started to work when I
did this. I'm honestly surprised that it did, and that I didn't have to
change a single line in the timing code past decoder (just something
minor to make external/cached text subtitles display). I committed it
immediately while avoiding thinking about it. But there really likely
are subtle problems of all sorts.
As far as I'm aware, gstreamer also supports backward playback. When I
looked at this years ago, I couldn't find a way to actually try this,
and I didn't revisit it now. Back then I also read talk slides from the
person who implemented it, and I'm not sure if and which ideas I might
have taken from it. It's possible that the timestamp reversal is
inspired by it, but I didn't check. (I think it claimed that it could
avoid large changes by changing a sign?)
VapourSynth has some sort of reverse function, which provides a backward
view on a video. The function itself is trivial to implement, as
VapourSynth aims to provide random access to video by frame numbers (so
you just request decreasing frame numbers). From what I remember, it
wasn't exactly fluid, but it worked. It's implemented by creating an
index, and seeking to the target on demand, and a bunch of caching. mpv
could use it, but it would either require using VapourSynth as demuxer
and decoder for everything, or replacing the current file every time
something is supposed to be played backwards.
FFmpeg's libavfilter has reversal filters for audio and video. These
require buffering the entire media data of the file, and don't really
fit into mpv's architecture. It could be used by playing a libavfilter
graph that also demuxes, but that's like VapourSynth but worse.
Let's blame FFmpeg for just overwriting the samplerate in
av_frame_copy_props(). Can't fully hide my own brain damage though,
since mp_aframe_config_copy() expected that the rate is copied (that
function also copies format and channel layout).
Move it from af_lavrresample.c to a new aconverter.c file, which is
independent from the filter chain code. It also doesn't use mp_audio,
and thus has no GPL dependencies.
Preparation for later commits. Not particularly well tested, so have
fun.
This is pretty pointless, but I believe it allows us to claim that the
new code is not affected by the copyright of the old code. This is
needed, because the original mp_audio struct was written by someone who
has disagreed with LGPL relicensing (it was called af_data at the time,
and was defined in af.h).
The "GPL'ed" struct contents that surive are pretty trivial: just the
data pointer, and some metadata like the format, samplerate, etc. - but
at least in this case, any new code would be extremely similar anyway,
and I'm not really sure whether it's OK to claim different copyright. So
what we do is we just use AVFrame (which of course is LGPL with 100%
certainty), and add some accessors around it to adapt it to mpv
conventions.
Also, this gets rid of some annoying conventions of mp_audio, like the
struct fields that require using an accessor to write to them anyway.
For the most part, this change is only dumb replacements of mp_audio
related functions and fields. One minor actual change is that you can't
allocate the new type on the stack anymore.
Some code still uses mp_audio. All audio filter code will be deleted, so
it makes no sense to convert this code. (Audio filters which are LGPL
and which we keep will have to be ported to a new filter infrastructure
anyway.) player/audio.c uses it because it interacts with the old filter
code. push.c has some complex use of mp_audio and mp_audio_buffer, but
this and pull.c will most likely be rewritten to do something else.
Niklas Haas
Jan Ekström
sfan5
wm4