Commit Graph

1921 Commits

Author SHA1 Message Date
Michael Niedermayer b6d9ecfe22 swscale/utils: Fix chrSrcHSubSample for GBRAP16
Fixes part of Ticket5264

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 67e5bd0c50)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-04-29 14:23:40 +02:00
Michael Niedermayer 32a453eae2 swscale/input: Fix GBRAP16 input
Fixes part of Ticket5264

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit df36257a53)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-04-29 14:23:40 +02:00
Michael Niedermayer 7bf84d1c2b swscale/x86/output: Fix yuv2planeX_16* with unaligned destination
Reviewed-by: BBB
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit f6492a2ea8)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-04-29 14:23:40 +02:00
Michael Niedermayer bd641685f4 swscale/x86/output: Move code into yuv2planeX_mainloop
Reviewed-by: BBB
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit d07f6e5f1c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-04-29 14:23:40 +02:00
Michael Niedermayer 372ea28f68 swscale/swscale: Add some sanity checks for srcSlice* parameters
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 321e85e176)

Conflicts:

	libswscale/swscale.c

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-31 00:24:27 +01:00
Michael Niedermayer c48296d3bf swscale/x86/rgb2rgb_template: Fix planar2x() for short width
Fixes: 451b3e0cf956c0bd2f27ed753ac24050/asan_heap-oob_2873c01_3231_7ed10a9464d15f0d57277f5917c566a8.AVI

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit c8a9aaab26)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-31 00:24:27 +01:00
Michael Niedermayer aea2f5a6ee swscale/swscale_unscaled: Fix odd height inputs for bayer_to_yv12_wrapper()
Fixes: 372d2df1f04b49e25f109f07f90b1505/asan_heap-oob_2835d2e_8501_99e0114d7ba3a6db885d0b4684d200c1.cine
Fixes out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 757248ea3c)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-31 00:24:27 +01:00
Michael Niedermayer 78c9e1f00b swscale/swscale_unscaled: Fix odd height inputs for bayer_to_rgb24_wrapper()
Fixes: 372d2df1f04b49e25f109f07f90b1505/asan_heap-oob_2835d2e_8501_99e0114d7ba3a6db885d0b4684d200c1.cine
Fixes out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit ad3b6fa7d8)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-31 00:24:27 +01:00
Michael Niedermayer 0f091f808a swscale/swscale-test: Fix slice height in random reference data creation.
Found-by: Pedro Arthur <bygrandao@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-31 00:24:12 +01:00
Michael Niedermayer f0b6705e1a swscale/utils: Detect and skip unneeded sws_setColorspaceDetails() calls
This avoids running various table inits unnecessarily

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit cc538e9dbd)

Conflicts:

	libswscale/utils.c
2016-01-15 17:23:55 +01:00
Michael Niedermayer 1ba90ce49e swscale/yuv2rgb: Increase YUV2RGB table headroom
This makes SWS more robust
Fixes: 07650a772d98aa63b0fed6370dc89037/asan_heap-oob_27ddeaf_2657_2c81ff264dee5d9712cb3251fb9c3bbb.264
Fixes: out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8f3a9a8c27)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 17:23:55 +01:00
Michael Niedermayer 4bafd67795 swscale/yuv2rgb: Factor YUVRGB_TABLE_LUMA_HEADROOM out
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 5e5f82a287)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 17:23:55 +01:00
Michael Niedermayer dbe194002a swscale/utils: Use normal bilinear scaler if fast cannot be used due to tiny dimensions
Fixes Ticket4960

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 1edf129cbc)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 17:23:54 +01:00
Michael Niedermayer bb70069e6a swscale/utils: Fix intermediate format for cascaded alpha downscaling
Fixes Ticket4926

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit b83d8be6bf)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-15 17:23:54 +01:00
Michael Niedermayer 0fa22a42d6 swscale/utils: Fix for runtime error: left shift of negative value -1
Fixes: c106b36fa36db8ff8f3ed0c82be7bea2/asan_heap-oob_32699f0_6321_467b9a1d7e03d7cfd310b7e65dc53bcc.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 325b59368d)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-20 10:59:27 +01:00
Michael Niedermayer 26e98f9ae5 swscale/x86/rgb2rgb_template: Fallback to mmx in interleaveBytes() if the alignment is insufficient for SSE*
This also as a sideeffect fixes the non aligned case

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a066ff89bc)
2015-12-15 11:18:39 +01:00
Michael Niedermayer 45430c3753 swscale/x86/rgb2rgb_template: Do not crash on misaligend stride
Fixes Ticket5013

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 80bfce35cc)
2015-12-15 11:18:37 +01:00
Michael Niedermayer fb47b6cb4b swscale/utils: Clear pix buffers
Fixes use of uninitialized memory
Fixes: a96874b9466b6edc660a519c7ad47977_signal_sigsegv_7ffff713351a_744_nc_sample.avi with memlimit 2147483648

Found-by: Samuel Groß, Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit a5d44d5c22)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-07-20 04:43:41 +02:00
Michael Niedermayer 49ad1658b2 swscale/swscale_unscaled: Fix rounding difference with RGBA output between little and big endian
Fixes fate/dds-rgb16 on big endian

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f6ab967eae)

Conflicts:

	tests/ref/fate/dds-rgb16
2015-07-20 04:43:40 +02:00
James Almer 48127e2ad6 swscale/x86/rgb2rgb_template: fix signedness of v in shuffle_bytes_2103_{mmx,mmxext}
Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit e22edbfd41)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-07-20 04:43:40 +02:00
James Almer 860dc1e19b swscale/x86/rgb2rgb_template: add missing xmm clobbers
Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: James Almer <jamrial@gmail.com>
(cherry picked from commit 910eeab480)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-07-20 04:43:40 +02:00
Michael Niedermayer e920af2cdf swscale/rgb2rgb_template: Fix signedness of v in shuffle_bytes_2103_c()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 7604358018)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-07-20 04:43:40 +02:00
Michael Niedermayer 12e166dacb swscale/rgb2rgb_template: Implement shuffle_bytes_0321_c and fix shuffle_bytes_2103_c on BE
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit abb833c568)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-07-20 04:43:40 +02:00
Michael Niedermayer 80cec4e496 swscale/rgb2rgb_template: Disable shuffle_bytes_2103_c on big endian
The function is specific to little endian

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4df3cf90bf)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-07-20 04:43:40 +02:00
Nick Lewycky a5167b4d66 libswscale/x86/hscale_fast_bilinear_simd.c: Include BX in the clobber list on x86_64, because it isn't implicitly included when PIC is on.
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 48e9f68384)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-15 10:04:51 +02:00
Rong Yan be1b665dec swscale/ppc/swscale_altivec.c: POWER LE support in yuv2planeX_8() delete macro GET_VF() it was wrong
GCC tool had a bug of PPC intrinsic interpret, which has been fixed in GCC 4.9.1. This bug lead to
errors in two of our previous patches. We found this when we update our GCC tools to 4.9.1 and by
reading the related info on GCC website. We fix our previous error in two separate commits

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 603c839398)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-05-14 19:07:33 +02:00
Michael Niedermayer ae20682f6b swscale: Add prefix to updateMMXDitherTables()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-27 19:23:46 +01:00
Michael Niedermayer 007498fc1a swscale/utils: clear formatConvBuffer on allocation
Fixes use of uninitialized memory
Fixes: asan_heap-oob_35ca682_1474_cov_3230122439_aletrek_tga_16bit.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-27 03:13:14 +01:00
Michael Niedermayer fb8e5044b4 swscale/utils: add asserts to check filterpos
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-24 01:00:24 +01:00
Michael Niedermayer 1895d414aa swscale/utils: More carefully merge and clear coefficients outside the input
Fixes out of array read
Fixes: asan_heap-oob_35ca682_1474_cov_3230122439_aletrek_tga_16bit.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-24 01:00:24 +01:00
Michael Niedermayer 03bffb68f6 swscale: Use av_clip_uintp2()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-21 14:49:17 +01:00
Michael Niedermayer b20426398c swscale/yuv2rgb: Use av_clip_uint8()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-21 14:44:47 +01:00
Michael Niedermayer d501b986a9 swscale/bayer_template: Add () to protect the argument of BAYER_READ()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-18 01:55:15 +01:00
Michael Niedermayer 2a1b79d7e6 Merge commit 'e51f22122d23589e93ac4f0b3e570bb925755915'
* commit 'e51f22122d23589e93ac4f0b3e570bb925755915':
  swscale: Check memory allocations

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-17 22:22:17 +01:00
Federico Tomassetti e51f22122d swscale: Check memory allocations
CC: libav-stable@libav.org
Bug-Id: CID 1267888 / CID 1267890
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2015-02-17 12:16:43 -05:00
Michael Niedermayer 692b22626e swscale/utils: Limit filter shifting so as not to read from prior the array
Fixes out of array read
Fixes: asan_heap-oob_1fb2f9b_3780_cov_3984375136_usf.mkv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-05 00:33:50 +01:00
Michael Niedermayer f30798584f swscale/input: fix rgba64 alpha non native
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-21 21:21:00 +01:00
Michael Niedermayer 95d04690aa swscale/input: Fix alpha of YA16 input
Fixes Ticket4278

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-21 21:21:00 +01:00
Vittorio Giovara 2df7277711 swscale: fix gbrap to gbrap alpha scaling 2015-01-16 02:47:15 +01:00
Vittorio Giovara 89df3fd49e swscale: fix warning about incompatible function pointer type 2015-01-16 02:47:10 +01:00
Michael Niedermayer 27513846f6 Merge commit '928061670e873e816daa14827853b7e11221ff5f'
* commit '928061670e873e816daa14827853b7e11221ff5f':
  libswscale: GBRAP input & output and GBRAP16 input support

Conflicts:
	libswscale/input.c
	libswscale/swscale_internal.h
	libswscale/swscale_unscaled.c
	libswscale/utils.c
	tests/ref/fate/filter-pixdesc-gbrap
	tests/ref/fate/filter-pixfmts-copy
	tests/ref/fate/filter-pixfmts-null
	tests/ref/fate/filter-pixfmts-scale
	tests/ref/fate/filter-pixfmts-vflip

See: 5c057433cc
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-15 05:47:25 +01:00
Paul B Mahol 928061670e libswscale: GBRAP input & output and GBRAP16 input support
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2015-01-14 17:17:24 +01:00
Michael Niedermayer 90de28befd Merge commit '1dd797e3c9f179f957316a0becbec048b42df8aa'
* commit '1dd797e3c9f179f957316a0becbec048b42df8aa':
  swscale: check memory allocations

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-06 21:44:50 +01:00
Vittorio Giovara 1dd797e3c9 swscale: check memory allocations
CC: libav-stable@libav.org
Bug-Id: CID 1257779
2015-01-05 15:35:39 +01:00
Kieran Kunhya 18982f084c swscale: Pass through chroma positions in sws_getCachedContext
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-26 17:55:12 +01:00
Michael Niedermayer 3a2bed1f9e swscale/yuv2rgb: Fix width % 4 != 0 with bgr4_byte/rgb4_byte
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-24 00:33:21 +01:00
Michael Niedermayer 3848512d48 swscale/yuv2rgb: Fix width % 4 != 0 with bgr4/rgb4
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-24 00:32:53 +01:00
Michael Niedermayer 3d00ba2688 swscale/yuv2rgb: fix width % 8 != 0 and rgb/bgr8 output
Fixes Ticket2570

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-24 00:23:49 +01:00
Michael Niedermayer 2a983ff7fe swscale: increase yuv2rgb table headroom
Fixes out of array access
Fixes: case2_bad_read_yuv2rgbx32.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-16 23:37:26 +01:00
Michael Niedermayer e827f65618 Merge commit '33c827f632f95ffe3399b695a5a0d47b366b6e20'
* commit '33c827f632f95ffe3399b695a5a0d47b366b6e20':
  swscale: Properly scale YUV

Conflicts:
	libswscale/swscale_unscaled.c
	tests/ref/fate/filter-pixdesc-yuv420p10be
	tests/ref/fate/filter-pixdesc-yuv420p10le
	tests/ref/fate/filter-pixdesc-yuv420p9be
	tests/ref/fate/filter-pixdesc-yuv420p9le
	tests/ref/fate/filter-pixdesc-yuva420p10be
	tests/ref/fate/filter-pixdesc-yuva420p10le
	tests/ref/fate/filter-pixdesc-yuva420p9be
	tests/ref/fate/filter-pixdesc-yuva420p9le
	tests/ref/fate/filter-pixfmts-copy
	tests/ref/fate/filter-pixfmts-null
	tests/ref/fate/filter-pixfmts-scale
	tests/ref/fate/filter-pixfmts-vflip

See: abe0b8e9f3
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-12-06 01:30:08 +01:00