demux: fix backward demuxing freeze if first packet is not a keyframe

Some files don't start with keyframe packets. Normally, this is not
sane, but the sample file which triggered this was a cut TV capture
transport stream. And this shouldn't happen anyway.

Introduce a further heuristic: if the last seek target was before the
start of the cached data, and the start of the cache is marked as BOF
(beginning of file), then we won't find anything better. This is
possibly a bit shaky, because both seek_start and back_seek_pos weren't
made for this purpose. But I can't come up with situations where this
would actually break. (Leave this to shitty broken files I hit later.)

I also considered finding the first packet in the cache that is marked
as keyframe, i.e. the first actual seek target, and comparing it to
"first", but I didn't like it much. Well whatever.

It's a bit silly that this caused a hard freeze (and similar issues
still will). The problem is that the demuxer holds the lock and has no
reason to release it. And in general, there's a single lock for the
entire demuxer cache. Finer grained locking would probably not make much
sense. In theory status of available data and maybe certain commands to
the demuxer could be moved to separate locks, but it would raise
complexity, and you'd probably still need to get the central lock in
some cases, which would deadlock you anyway.

It would still be nice if some minor corner case in the wonderfully
terrible and complex backward demuxer state machine couldn't lock up the
player. As a hack, unlock and then immediately lock again. Depending on
the OS mutex implementation, this may give other waiters a chance to
grab the lock. This is not a guarantee (some OSes may for example not
wake up other waiters until the next time slice or something), but works
well on Linux.

demux/demux.c

@@ -1259,6 +1259,10 @@ static void perform_backward_seek(struct demux_internal *in)
     MP_VERBOSE(in, "triggering backward seek to get more packets\n");
     queue_seek(in, target, SEEK_SATAN | SEEK_HR, false);
     in->reading = true;
+
+    // Don't starve other threads.
+    pthread_mutex_unlock(&in->lock);
+    pthread_mutex_lock(&in->lock);
 }
 
 // Search for a packet to resume demuxing from.
@@ -1393,7 +1397,10 @@ static void find_backward_restart_pos(struct demux_stream *ds)
     }
 
     if (!target) {
-        if (ds->queue->is_bof && first == ds->queue->head) {
+        if (ds->queue->is_bof &&
+                (first == ds->queue->head ||
+                 ds->back_seek_pos < ds->queue->seek_start))
+        {
             MP_VERBOSE(in, "BOF for stream %d\n", ds->index);
             ds->back_restarting = false;
             ds->back_range_started = false;