core: fix crash when video filter returns inf as PTS

When a video filter returned inf as PTS, the player crashed. One
reason for this was that decode_audio() was called with a negative
minlen parameter, which at some point caused it to call a memory
allocation function with a ridiculous value, triggering an out of
memory code path in talloc.c. (talloc.c has been modified to abort()
on out of memory situations.)

Fix this by sanity checking minlen in decode_audio(). (The check
against outbuf->len always succeeded, because it's an unsigned
comparison.)

Make an existing sanity check in mplayer.c more robust: check for NaN
too, which happens if the video PTS is inf.

This happened with "-vf pullup,softpulldown" (but is not triggered when
the following commit is applied).

audio/decode/dec_audio.c

@@ -405,7 +405,7 @@ int decode_audio(sh_audio_t *sh_audio, struct bstr *outbuf, int minlen)
         return -1;
     max_decode_len -= max_decode_len % unitsize;
 
-    while (outbuf->len < minlen) {
+    while (minlen >=0 && outbuf->len < minlen) {
 	int declen = (minlen - outbuf->len) / filter_multiplier
 	    + (unitsize << 5); // some extra for possible filter buffering
 	if (huge_filter_buffer)

core/mplayer.c

@@ -2124,7 +2124,7 @@ static int audio_start_sync(struct MPContext *mpctx, int playsize)
             bytes = 0;
         }
 
-        if (fabs(ptsdiff) > 300)   // pts reset or just broken?
+        if (fabs(ptsdiff) > 300 || isnan(ptsdiff))   // pts reset or just broken?
             bytes = 0;
 
         if (bytes > 0)