RepoMirrors/mpv
1
0
Fork 0
mirror of https://github.com/mpv-player/mpv synced 2025-02-07 23:51:49 +00:00
Code Issues Releases Wiki Activity
4005cda614
mpv/video/out/opengl/hwdec_cuda.c

676 lines
19 KiB
C
Raw Normal View History

hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
/*
* Copyright (c) 2016 Philip Langdale <philipl@overt.org>
*
* This file is part of mpv.
*
* mpv is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* mpv is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with mpv. If not, see <http://www.gnu.org/licenses/>.
*/
/*
* This hwdec implements an optimized output path using CUDA->OpenGL
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
* or CUDA->Vulkan interop for frame data that is stored in CUDA
* device memory. Although it is not explicit in the code here, the
* only practical way to get data in this form is from the
* nvdec/cuvid decoder.
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
*/
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#include <unistd.h>
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
#include <ffnvcodec/dynlink_loader.h>
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
#include <libavutil/hwcontext.h>
#include <libavutil/hwcontext_cuda.h>
vo_opengl: refactor into vo_gpu This is done in several steps: 1. refactor MPGLContext -> struct ra_ctx 2. move GL-specific stuff in vo_opengl into opengl/context.c 3. generalize context creation to support other APIs, and add --gpu-api 4. rename all of the --opengl- options that are no longer opengl-specific 5. move all of the stuff from opengl/* that isn't GL-specific into gpu/ (note: opengl/gl_utils.h became opengl/utils.h) 6. rename vo_opengl to vo_gpu 7. to handle window screenshots, the short-term approach was to just add it to ra_swchain_fns. Long term (and for vulkan) this has to be moved to ra itself (and vo_gpu altered to compensate), but this was a stop-gap measure to prevent this commit from getting too big 8. move ra->fns->flush to ra_gl_ctx instead 9. some other minor changes that I've probably already forgotten Note: This is one half of a major refactor, the other half of which is provided by rossy's following commit. This commit enables support for all linux platforms, while his version enables support for all non-linux platforms. Note 2: vo_opengl_cb.c also re-uses ra_gl_ctx so it benefits from the --opengl- options like --opengl-early-flush, --opengl-finish etc. Should be a strict superset of the old functionality. Disclaimer: Since I have no way of compiling mpv on all platforms, some of these ports were done blindly. Specifically, the blind ports included context_mali_fbdev.c and context_rpi.c. Since they're both based on egl_helpers, the port should have gone smoothly without any major changes required. But if somebody complains about a compile error on those platforms (assuming anybody actually uses them), you know where to complain.
2017-09-14 06:04:55 +00:00
#include "video/out/gpu/hwdec.h"
vo_gpu: hwdec_cuda: Guard GL and Vulkan headers properly We are currently unnecessarily including vulkan headers even when not building with vulkan support. I also guarded the GL header inclusion even though this doesn't appear to break anything today. Fixes #6330.
2018-11-18 21:29:42 +00:00
#include "video/out/gpu/utils.h"
vo_opengl: hwdec_cuda: add yuv420p support Because it allows easier testing of filters + hwdec. Make the texture setup code a bit more generic so it doesn't get too much of a mess. We also use the GL renderer utility function gl_find_unorm_format(), which saves us additional work with OpenGL's semi-redundant format specifiers.
2017-01-16 14:39:38 +00:00
#include "formats.h"
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
#include "options/m_config.h"
vo_gpu: hwdec_cuda: Guard GL and Vulkan headers properly We are currently unnecessarily including vulkan headers even when not building with vulkan support. I also guarded the GL header inclusion even though this doesn't appear to break anything today. Fixes #6330.
2018-11-18 21:29:42 +00:00
#if HAVE_GL
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
#include "ra_gl.h"
vo_gpu: hwdec_cuda: Guard GL and Vulkan headers properly We are currently unnecessarily including vulkan headers even when not building with vulkan support. I also guarded the GL header inclusion even though this doesn't appear to break anything today. Fixes #6330.
2018-11-18 21:29:42 +00:00
#endif
#if HAVE_VULKAN
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
#include "video/out/placebo/ra_pl.h"
vo_gpu: hwdec_cuda: Guard GL and Vulkan headers properly We are currently unnecessarily including vulkan headers even when not building with vulkan support. I also guarded the GL header inclusion even though this doesn't appear to break anything today. Fixes #6330.
2018-11-18 21:29:42 +00:00
#endif
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#if HAVE_WIN32_DESKTOP
#include <versionhelpers.h>
#endif
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
struct priv_owner {
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
struct mp_hwdec_ctx hwctx;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CudaFunctions *cu;
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
CUcontext display_ctx;
CUcontext decode_ctx;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
bool is_gl;
bool is_vk;
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
enum pl_handle_type handle_type;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
};
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
struct ext_vk {
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
CUexternalMemory mem;
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
CUmipmappedArray mma;
const struct pl_tex *pltex;
const struct pl_sync *sync;
CUexternalSemaphore ss;
CUexternalSemaphore ws;
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
};
struct priv {
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
struct mp_image layout;
vo_opengl: hwdec_cuda: add yuv420p support Because it allows easier testing of filters + hwdec. Make the texture setup code a bit more generic so it doesn't get too much of a mess. We also use the GL renderer utility function gl_find_unorm_format(), which saves us additional work with OpenGL's semi-redundant format specifiers.
2017-01-16 14:39:38 +00:00
CUgraphicsResource cu_res[4];
CUarray cu_array[4];
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
CUcontext display_ctx;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
struct ext_vk evk[4];
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
};
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
static int check_cu(struct ra_hwdec *hw, CUresult err, const char *func)
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
{
const char *err_name;
const char *err_string;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
struct priv_owner *p = hw->priv;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
MP_TRACE(hw, "Calling %s\n", func);
if (err == CUDA_SUCCESS)
return 0;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
p->cu->cuGetErrorName(err, &err_name);
p->cu->cuGetErrorString(err, &err_string);
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
MP_ERR(hw, "%s failed", func);
if (err_name && err_string)
MP_ERR(hw, " -> %s: %s", err_name, err_string);
MP_ERR(hw, "\n");
return -1;
}
#define CHECK_CU(x) check_cu(hw, (x), #x)
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
static int cuda_init(struct ra_hwdec *hw)
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
{
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
CUdevice display_dev;
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
AVBufferRef *hw_device_ctx = NULL;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
CUcontext dummy;
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
int ret = 0;
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
struct priv_owner *p = hw->priv;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CudaFunctions *cu;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#if HAVE_GL
p->is_gl = ra_is_gl(hw->ra);
if (p->is_gl) {
GL *gl = ra_gl_get(hw->ra);
if (gl->version < 210 && gl->es < 300) {
MP_VERBOSE(hw, "need OpenGL >= 2.1 or OpenGL-ES >= 3.0\n");
return -1;
}
}
#endif
#if HAVE_VULKAN
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
p->handle_type =
#if HAVE_WIN32_DESKTOP
IsWindows8OrGreater() ? PL_HANDLE_WIN32 : PL_HANDLE_WIN32_KMT;
#else
PL_HANDLE_FD;
#endif
const struct pl_gpu *gpu = ra_pl_get(hw->ra);
p->is_vk = gpu != NULL;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
if (p->is_vk) {
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
if (!(gpu->export_caps.tex & p->handle_type)) {
MP_VERBOSE(hw, "CUDA hwdec with Vulkan requires exportable texture memory of type 0x%X.\n",
p->handle_type);
return -1;
} else if (!(gpu->export_caps.sync & p->handle_type)) {
MP_VERBOSE(hw, "CUDA hwdec with Vulkan requires exportable semaphores of type 0x%X.\n",
p->handle_type);
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
return -1;
}
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
}
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#endif
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
if (!p->is_gl && !p->is_vk) {
MP_VERBOSE(hw, "CUDA hwdec only works with OpenGL or Vulkan backends.\n");
return -1;
}
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
ret = cuda_load_functions(&p->cu, NULL);
if (ret != 0) {
vo_opengl: hwdec_cuda: make some init errors verbose Improves autoprobe behavior. This is equivalent to other hwdec interop wrappers. If CUDA is just not available, it should remain silent.
2016-11-24 17:14:55 +00:00
MP_VERBOSE(hw, "Failed to load CUDA symbols\n");
vo_opengl: hwdec_cuda: fix crash when trying to use hwdec=cuda if cuda SDK is not present If CUDA SDK wasn't installed, mpv crashed immediately with the message "Failed to load CUDA symbols"
2016-11-24 12:11:44 +00:00
return -1;
vo_opengl: hwdec_cuda: Use dynamic loading for cuda functions This change applies the pattern used in ffmpeg to dynamically load cuda, to avoid requiring the CUDA SDK at build time.
2016-10-08 23:51:15 +00:00
}
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
cu = p->cu;
vo_opengl: hwdec_cuda: Use dynamic loading for cuda functions This change applies the pattern used in ffmpeg to dynamically load cuda, to avoid requiring the CUDA SDK at build time.
2016-10-08 23:51:15 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
if (p->is_vk && !cu->cuImportExternalMemory) {
MP_ERR(hw, "CUDA hwdec with Vulkan requires driver version 410.48 or newer.\n");
return -1;
}
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
ret = CHECK_CU(cu->cuInit(0));
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
if (ret < 0)
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
// Allocate display context
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
if (p->is_gl) {
unsigned int device_count;
ret = CHECK_CU(cu->cuGLGetDevices(&device_count, &display_dev, 1,
CU_GL_DEVICE_LIST_ALL));
if (ret < 0)
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
ret = CHECK_CU(cu->cuCtxCreate(&p->display_ctx, CU_CTX_SCHED_BLOCKING_SYNC,
display_dev));
if (ret < 0)
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
p->decode_ctx = p->display_ctx;
int decode_dev_idx = -1;
mp_read_option_raw(hw->global, "cuda-decode-device", &m_option_type_choice,
&decode_dev_idx);
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
if (decode_dev_idx > -1) {
CUdevice decode_dev;
ret = CHECK_CU(cu->cuDeviceGet(&decode_dev, decode_dev_idx));
if (ret < 0)
goto error;
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
if (decode_dev != display_dev) {
MP_INFO(hw, "Using separate decoder and display devices\n");
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
// Pop the display context. We won't use it again during init()
ret = CHECK_CU(cu->cuCtxPopCurrent(&dummy));
if (ret < 0)
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
ret = CHECK_CU(cu->cuCtxCreate(&p->decode_ctx, CU_CTX_SCHED_BLOCKING_SYNC,
decode_dev));
if (ret < 0)
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
}
}
} else if (p->is_vk) {
#if HAVE_VULKAN
int count;
ret = CHECK_CU(cu->cuDeviceGetCount(&count));
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
if (ret < 0)
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
display_dev = -1;
for (int i = 0; i < count; i++) {
CUdevice dev;
ret = CHECK_CU(cu->cuDeviceGet(&dev, i));
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
if (ret < 0)
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
continue;
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
CUuuid uuid;
ret = CHECK_CU(cu->cuDeviceGetUuid(&uuid, dev));
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
if (ret < 0)
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
continue;
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
if (memcmp(gpu->uuid, uuid.bytes, sizeof (gpu->uuid)) == 0) {
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
display_dev = dev;
break;
}
}
if (display_dev == -1) {
MP_ERR(hw, "Could not match Vulkan display device in CUDA.\n");
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
}
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
ret = CHECK_CU(cu->cuCtxCreate(&p->display_ctx, CU_CTX_SCHED_BLOCKING_SYNC,
display_dev));
if (ret < 0)
vo_gpu: hwdec_cuda: Clean up init() error handling Currently, the error paths in init() are a bit confusing, and we can end up trying to pop the current context when there is no context, which leads to distracting error messages. I also added an explicit path to return early if the GPU backend is not OpenGL or Vulkan. It's pointless to do any other cuda init after that point. (Of course, someone could write more interops.) Fixes #6256
2018-10-28 16:25:30 +00:00
return -1;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
p->decode_ctx = p->display_ctx;
#endif
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
}
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
hw_device_ctx = av_hwdevice_ctx_alloc(AV_HWDEVICE_TYPE_CUDA);
if (!hw_device_ctx)
goto error;
AVHWDeviceContext *device_ctx = (void *)hw_device_ctx->data;
AVCUDADeviceContext *device_hwctx = device_ctx->hwctx;
vo_opengl: hwdec_cuda: Support separate decode and display devices In a multi GPU scenario, it may be desirable to use different GPUs for decode and display responsibilities. For example, if a secondary GPU has better video decoding capabilities. In such a scenario, we need to initialise a separate context for each GPU, and use the display context in hwdec_cuda, while passing the decode context to avcodec. Once that's done, the actually hand-off between the two GPUs is transparent to us (It happens during the cuMemcpy2D operation which copies the decoded frame from a cuda buffer to the OpenGL texture). In the end, the bulk of the work is around introducing a new configuration option to specify the decode device.
2017-05-29 16:48:10 +00:00
device_hwctx->cuda_ctx = p->decode_ctx;
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
ret = av_hwdevice_ctx_init(hw_device_ctx);
if (ret < 0) {
MP_ERR(hw, "av_hwdevice_ctx_init failed\n");
goto error;
}
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
ret = CHECK_CU(cu->cuCtxPopCurrent(&dummy));
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
if (ret < 0)
goto error;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
p->hwctx = (struct mp_hwdec_ctx) {
hwdec: don't require setting legacy hwdec fields With the recent changes, mpv's internal mechanisms got synced to libavcodec's once more. Some things are still needed for filters (until the mechanism gets replaced), but there's no need to require other hwdec methods to use these fields. So remove them where they are unnecessary. Also fix some minor leaks in the dxva2 backends, and set the driver_name field in the Apple ones. Untested on Apple crap.
2017-12-02 02:58:04 +00:00
.driver_name = hw->driver->name,
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
.av_device_ref = hw_device_ctx,
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
};
hwdec_devices_add(hw->devs, &p->hwctx);
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
return 0;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
error:
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
av_buffer_unref(&hw_device_ctx);
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CHECK_CU(cu->cuCtxPopCurrent(&dummy));
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: hwdec_cuda: export AVHWDeviceContext So we can use it for filtering later.
2017-01-16 14:34:34 +00:00
return -1;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
}
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
static void cuda_uninit(struct ra_hwdec *hw)
{
struct priv_owner *p = hw->priv;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CudaFunctions *cu = p->cu;
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
hwdec: don't require setting legacy hwdec fields With the recent changes, mpv's internal mechanisms got synced to libavcodec's once more. Some things are still needed for filters (until the mechanism gets replaced), but there's no need to require other hwdec methods to use these fields. So remove them where they are unnecessary. Also fix some minor leaks in the dxva2 backends, and set the driver_name field in the Apple ones. Untested on Apple crap.
2017-12-02 02:58:04 +00:00
hwdec_devices_remove(hw->devs, &p->hwctx);
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
av_buffer_unref(&p->hwctx.av_device_ref);
if (p->decode_ctx && p->decode_ctx != p->display_ctx)
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CHECK_CU(cu->cuCtxDestroy(p->decode_ctx));
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
if (p->display_ctx)
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CHECK_CU(cu->cuCtxDestroy(p->display_ctx));
cuda_free_functions(&p->cu);
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
}
#undef CHECK_CU
#define CHECK_CU(x) check_cu((mapper)->owner, (x), #x)
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#if HAVE_VULKAN
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
static bool cuda_ext_vk_init(struct ra_hwdec_mapper *mapper,
const struct ra_format *format, int n)
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
{
struct priv_owner *p_owner = mapper->owner->priv;
struct priv *p = mapper->priv;
CudaFunctions *cu = p_owner->cu;
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
int mem_fd = -1, wait_fd = -1, signal_fd = -1;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
int ret = 0;
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
struct ext_vk *evk = &p->evk[n];
const struct pl_gpu *gpu = ra_pl_get(mapper->ra);
struct pl_tex_params tex_params = {
.w = mp_image_plane_w(&p->layout, n),
.h = mp_image_plane_h(&p->layout, n),
.d = 0,
.format = ra_pl_fmt_get(format),
.sampleable = true,
.sample_mode = format->linear_filter ? PL_TEX_SAMPLE_LINEAR
: PL_TEX_SAMPLE_NEAREST,
.export_handle = p_owner->handle_type,
};
evk->pltex = pl_tex_create(gpu, &tex_params);
if (!evk->pltex) {
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
goto error;
}
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
struct ra_tex *ratex = talloc_ptrtype(NULL, ratex);
ret = mppl_wrap_tex(mapper->ra, evk->pltex, ratex);
if (!ret) {
pl_tex_destroy(gpu, &evk->pltex);
talloc_free(ratex);
goto error;
}
mapper->tex[n] = ratex;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
#if !HAVE_WIN32_DESKTOP
if (evk->pltex->params.export_handle == PL_HANDLE_FD) {
mem_fd = dup(evk->pltex->shared_mem.handle.fd);
if (mem_fd < 0) {
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
goto error;
}
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
}
#endif
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
CUDA_EXTERNAL_MEMORY_HANDLE_DESC ext_desc = {
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#if HAVE_WIN32_DESKTOP
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
.type = IsWindows8OrGreater()
? CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32
: CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT,
.handle.win32.handle = evk->pltex->shared_mem.handle.handle,
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#else
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
.type = CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD,
.handle.fd = mem_fd,
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#endif
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
.size = evk->pltex->shared_mem.size,
.flags = 0,
};
ret = CHECK_CU(cu->cuImportExternalMemory(&evk->mem, &ext_desc));
if (ret < 0)
goto error;
// CUDA takes ownership of imported memory
mem_fd = -1;
CUarray_format cufmt;
switch (format->pixel_size / format->num_components) {
case 1:
cufmt = CU_AD_FORMAT_UNSIGNED_INT8;
break;
case 2:
cufmt = CU_AD_FORMAT_UNSIGNED_INT16;
break;
default:
ret = -1;
goto error;
}
CUDA_EXTERNAL_MEMORY_MIPMAPPED_ARRAY_DESC tex_desc = {
.offset = evk->pltex->shared_mem.offset,
.arrayDesc = {
.Width = mp_image_plane_w(&p->layout, n),
.Height = mp_image_plane_h(&p->layout, n),
.Depth = 0,
.Format = cufmt,
.NumChannels = format->num_components,
.Flags = 0,
},
.numLevels = 1,
};
ret = CHECK_CU(cu->cuExternalMemoryGetMappedMipmappedArray(&evk->mma, evk->mem, &tex_desc));
if (ret < 0)
goto error;
ret = CHECK_CU(cu->cuMipmappedArrayGetLevel(&p->cu_array[n], evk->mma, 0));
if (ret < 0)
goto error;
evk->sync = pl_sync_create(gpu, p_owner->handle_type);
if (!evk->sync) {
ret = -1;
goto error;
}
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
#if !HAVE_WIN32_DESKTOP
if (evk->sync->handle_type == PL_HANDLE_FD) {
wait_fd = dup(evk->sync->wait_handle.fd);
signal_fd = dup(evk->sync->signal_handle.fd);
}
#endif
CUDA_EXTERNAL_SEMAPHORE_HANDLE_DESC w_desc = {
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#if HAVE_WIN32_DESKTOP
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
.type = IsWindows8OrGreater()
? CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32
: CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT,
.handle.win32.handle = evk->sync->wait_handle.handle,
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#else
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
.type = CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD,
.handle.fd = wait_fd,
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#endif
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
};
ret = CHECK_CU(cu->cuImportExternalSemaphore(&evk->ws, &w_desc));
if (ret < 0)
goto error;
wait_fd = -1;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
CUDA_EXTERNAL_SEMAPHORE_HANDLE_DESC s_desc = {
#if HAVE_WIN32_DESKTOP
.type = IsWindows8OrGreater()
? CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32
: CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_WIN32_KMT,
.handle.win32.handle = evk->sync->signal_handle.handle,
#else
.type = CU_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD,
.handle.fd = signal_fd,
#endif
};
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
ret = CHECK_CU(cu->cuImportExternalSemaphore(&evk->ss, &s_desc));
if (ret < 0)
goto error;
signal_fd = -1;
return true;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
error:
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
MP_ERR(mapper, "cuda_ext_vk_init failed\n");
if (mem_fd > -1)
close(mem_fd);
if (wait_fd > -1)
close(wait_fd);
if (signal_fd > -1)
close(signal_fd);
return false;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
}
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
static void cuda_ext_vk_uninit(struct ra_hwdec_mapper *mapper, int n)
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
{
struct priv_owner *p_owner = mapper->owner->priv;
struct priv *p = mapper->priv;
CudaFunctions *cu = p_owner->cu;
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
struct ext_vk *evk = &p->evk[n];
if (evk) {
if (evk->mma) {
CHECK_CU(cu->cuMipmappedArrayDestroy(evk->mma));
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
}
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
if (evk->mem) {
CHECK_CU(cu->cuDestroyExternalMemory(evk->mem));
}
if (evk->ss) {
CHECK_CU(cu->cuDestroyExternalSemaphore(evk->ss));
}
if (evk->ws) {
CHECK_CU(cu->cuDestroyExternalSemaphore(evk->ws));
}
pl_sync_destroy(ra_pl_get(mapper->ra), &evk->sync);
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
}
}
#endif // HAVE_VULKAN
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
static int mapper_init(struct ra_hwdec_mapper *mapper)
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
{
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
struct priv_owner *p_owner = mapper->owner->priv;
struct priv *p = mapper->priv;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
CUcontext dummy;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CudaFunctions *cu = p_owner->cu;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
int ret = 0, eret = 0;
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
p->display_ctx = p_owner->display_ctx;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
int imgfmt = mapper->src_params.hw_subfmt;
mapper->dst_params = mapper->src_params;
mapper->dst_params.imgfmt = imgfmt;
mapper->dst_params.hw_subfmt = 0;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
mp_image_set_params(&p->layout, &mapper->dst_params);
struct ra_imgfmt_desc desc;
if (!ra_get_imgfmt_desc(mapper->ra, imgfmt, &desc)) {
MP_ERR(mapper, "Unsupported format: %s\n", mp_imgfmt_to_name(imgfmt));
vo_opengl: hwdec_cuda: Support P016 output surfaces The latest 375.xx nvidia drivers add support for P016 output surfaces. In combination with an ffmpeg change to return those surfaces, we can display them. The bulk of the work is related to knowing which format you're dealing with at the right time. Once you know, it's straight forward.
2016-11-19 21:57:23 +00:00
return -1;
}
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
ret = CHECK_CU(cu->cuCtxPushCurrent(p->display_ctx));
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
if (ret < 0)
return ret;
vo_opengl: hwdec_cuda: use new format setup function Gives us automatically support for all formats vo_opengl supports.
2017-02-17 15:39:10 +00:00
for (int n = 0; n < desc.num_planes; n++) {
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
const struct ra_format *format = desc.planes[n];
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
if (p_owner->is_gl) {
#if HAVE_GL
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
struct ra_tex_params params = {
.dimensions = 2,
.w = mp_image_plane_w(&p->layout, n),
.h = mp_image_plane_h(&p->layout, n),
.d = 1,
.format = format,
.render_src = true,
.src_linear = format->linear_filter,
};
mapper->tex[n] = ra_tex_create(mapper->ra, &params);
if (!mapper->tex[n]) {
ret = -1;
goto error;
}
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
GLuint texture;
GLenum target;
ra_gl_get_raw_tex(mapper->ra, mapper->tex[n], &texture, &target);
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
ret = CHECK_CU(cu->cuGraphicsGLRegisterImage(&p->cu_res[n], texture, target,
CU_GRAPHICS_REGISTER_FLAGS_WRITE_DISCARD));
if (ret < 0)
goto error;
vo_opengl: hwdec_cuda: directly map GL textures and skip using PBOs The documentation around this stuff is poor, but I found an nvidia sample that demonstrates how to use the interop API most efficiently. (https://github.com/nvpro-samples/gl_cuda_interop_pingpong_st) Key lessons are: 1) you can register the texture itself and have cuda write to it, thereby skipping an additional copy through the PBO. 2) You don't have to be mapped when you do the copy - once you get a mapped pointer, it remains valid. Magic! This lets us throw out the PBOs as well as much of the explicit alignment and stride handling. CPU usage is slightly (~3%) lower for 4K content in one test case, so it makes a detectable difference, and presumably saves memory.
2016-09-23 04:39:40 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
ret = CHECK_CU(cu->cuGraphicsMapResources(1, &p->cu_res[n], 0));
if (ret < 0)
goto error;
vo_opengl: hwdec_cuda: directly map GL textures and skip using PBOs The documentation around this stuff is poor, but I found an nvidia sample that demonstrates how to use the interop API most efficiently. (https://github.com/nvpro-samples/gl_cuda_interop_pingpong_st) Key lessons are: 1) you can register the texture itself and have cuda write to it, thereby skipping an additional copy through the PBO. 2) You don't have to be mapped when you do the copy - once you get a mapped pointer, it remains valid. Magic! This lets us throw out the PBOs as well as much of the explicit alignment and stride handling. CPU usage is slightly (~3%) lower for 4K content in one test case, so it makes a detectable difference, and presumably saves memory.
2016-09-23 04:39:40 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
ret = CHECK_CU(cu->cuGraphicsSubResourceGetMappedArray(&p->cu_array[n], p->cu_res[n],
0, 0));
if (ret < 0)
goto error;
hwdec_cuda: Use the non-deprecated CUDA-GL interop API The nvidia examples use the old (as in CUDA 3.x) interop API which is deprecated, and I think not even functional on recent versions of CUDA for windows. As I was following the examples, I used this old API. So, let's update to the new API, and hopefully, it'll start working on windows too.
2016-09-10 03:07:43 +00:00
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
ret = CHECK_CU(cu->cuGraphicsUnmapResources(1, &p->cu_res[n], 0));
if (ret < 0)
goto error;
#endif
} else if (p_owner->is_vk) {
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
#if HAVE_VULKAN
ret = cuda_ext_vk_init(mapper, format, n);
if (ret < 0)
goto error;
#endif
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
}
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
}
error:
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
eret = CHECK_CU(cu->cuCtxPopCurrent(&dummy));
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
if (eret < 0)
return eret;
return ret;
}
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
static void mapper_uninit(struct ra_hwdec_mapper *mapper)
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
{
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
struct priv *p = mapper->priv;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
struct priv_owner *p_owner = mapper->owner->priv;
CudaFunctions *cu = p_owner->cu;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
CUcontext dummy;
// Don't bail if any CUDA calls fail. This is all best effort.
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CHECK_CU(cu->cuCtxPushCurrent(p->display_ctx));
vo_opengl: hwdec_cuda: add yuv420p support Because it allows easier testing of filters + hwdec. Make the texture setup code a bit more generic so it doesn't get too much of a mess. We also use the GL renderer utility function gl_find_unorm_format(), which saves us additional work with OpenGL's semi-redundant format specifiers.
2017-01-16 14:39:38 +00:00
for (int n = 0; n < 4; n++) {
hwdec_cuda: Use the non-deprecated CUDA-GL interop API The nvidia examples use the old (as in CUDA 3.x) interop API which is deprecated, and I think not even functional on recent versions of CUDA for windows. As I was following the examples, I used this old API. So, let's update to the new API, and hopefully, it'll start working on windows too.
2016-09-10 03:07:43 +00:00
if (p->cu_res[n] > 0)
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CHECK_CU(cu->cuGraphicsUnregisterResource(p->cu_res[n]));
vo_opengl: hwdec_cuda: add yuv420p support Because it allows easier testing of filters + hwdec. Make the texture setup code a bit more generic so it doesn't get too much of a mess. We also use the GL renderer utility function gl_find_unorm_format(), which saves us additional work with OpenGL's semi-redundant format specifiers.
2017-01-16 14:39:38 +00:00
p->cu_res[n] = 0;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#if HAVE_VULKAN
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
cuda_ext_vk_uninit(mapper, n);
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
#endif
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
ra_tex_free(mapper->ra, &mapper->tex[n]);
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
}
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
CHECK_CU(cu->cuCtxPopCurrent(&dummy));
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
}
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
static void mapper_unmap(struct ra_hwdec_mapper *mapper)
{
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
}
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
static int mapper_map(struct ra_hwdec_mapper *mapper)
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
{
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
struct priv *p = mapper->priv;
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
struct priv_owner *p_owner = mapper->owner->priv;
CudaFunctions *cu = p_owner->cu;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
CUcontext dummy;
int ret = 0, eret = 0;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
bool is_vk = p_owner->is_vk;
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
ret = CHECK_CU(cu->cuCtxPushCurrent(p->display_ctx));
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
if (ret < 0)
return ret;
vo_opengl: hwdec_cuda: use new format setup function Gives us automatically support for all formats vo_opengl supports.
2017-02-17 15:39:10 +00:00
for (int n = 0; n < p->layout.num_planes; n++) {
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
#if HAVE_VULKAN
if (is_vk) {
ret = pl_tex_export(ra_pl_get(mapper->ra),
p->evk[n].pltex, p->evk[n].sync);
if (!ret)
goto error;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
CUDA_EXTERNAL_SEMAPHORE_WAIT_PARAMS wp = { 0, };
ret = CHECK_CU(cu->cuWaitExternalSemaphoresAsync(&p->evk[n].ws,
&wp, 1, 0));
if (ret < 0)
goto error;
}
#endif
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
CUDA_MEMCPY2D cpy = {
.srcMemoryType = CU_MEMORYTYPE_DEVICE,
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
.srcDevice = (CUdeviceptr)mapper->src->planes[n],
.srcPitch = mapper->src->stride[n],
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
.srcY = 0,
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
.dstMemoryType = CU_MEMORYTYPE_ARRAY,
.dstArray = p->cu_array[n],
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
.WidthInBytes = mp_image_plane_w(&p->layout, n) *
mapper->tex[n]->params.format->pixel_size,
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
.Height = mp_image_plane_h(&p->layout, n),
};
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
ret = CHECK_CU(cu->cuMemcpy2DAsync(&cpy, 0));
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
if (ret < 0)
goto error;
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
#if HAVE_VULKAN
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
if (is_vk) {
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
CUDA_EXTERNAL_SEMAPHORE_SIGNAL_PARAMS sp = { 0, };
ret = CHECK_CU(cu->cuSignalExternalSemaphoresAsync(&p->evk[n].ss,
&sp, 1, 0));
if (ret < 0)
goto error;
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
}
vo_gpu: hwdec_cuda: Implement interop for placebo This change updates the vulkan interop code to work with the libplacebo based ra_vk, but also introduces direct VkImage sharing to avoid the use of the intermediate buffer. It is also necessary and desirable to introduce explicit semaphore bsed synchronisation for operations on the shared images. Synchronisation means we can safely reuse the same VkImage for every mapped frame, by ensuring the frame is copied to the VkImage before mapping the next frame. This functionality requires a 417.xx or newer nvidia driver, due to bugs in the VkImage interop in the earlier 411 and 415 drivers. It's definitely worth the effort, as the raw throughput is about twice that of implementation using an intermediate buffer.
2018-11-27 02:04:08 +00:00
#endif
vo_gpu: vulkan: hwdec_cuda: Add support for Vulkan interop Despite their place in the tree, hwdecs can be loaded and used just fine by the vulkan GPU backend. In this change we add Vulkan interop support to the cuda/nvdec hwdec. The overall process is mostly straight forward, so the main observation here is that I had to implement it using an intermediate Vulkan buffer because the direct VkImage usage is blocked by a bug in the nvidia driver. When that gets fixed, I will revist this. Nevertheless, the intermediate buffer copy is very cheap as it's all device memory from start to finish. Overall CPU utilisiation is pretty much the same as with the OpenGL GPU backend. Note that we cannot use a single intermediate buffer - rather there is a pool of them. This is done because the cuda memcpys are not explicitly synchronised with the texture uploads. In the basic case, this doesn't matter because the hwdec is not asked to map and copy the next frame until after the previous one is rendered. In the interpolation case, we need extra future frames available immediately, so we'll be asked to map/copy those frames and vulkan will be asked to render them. So far, harmless right? No. All the vulkan rendering, including the upload steps, are batched together and end up running very asynchronously from the CUDA copies. The end result is that all the copies happen one after another, and only then do the uploads happen, which means all textures are uploaded the same, final, frame data. Whoops. Unsurprisingly this results in the jerky motion because every 3/4 frames are identical. The buffer pool ensures that we do not overwrite a buffer that is still waiting to be uploaded. The ra_buf_pool implementation automatically checks if existing buffers are available for use and only creates a new one if it really has to. It's hard to say for sure what the maximum number of buffers might be but we believe it won't be so large as to make this strategy unusable. The highest I've seen is 12 when using interpolation with tscale=bicubic. A future optimisation here is to synchronise the CUDA copies with respect to the vulkan uploads. This can be done with shared semaphores that would ensure the copy of the second frames only happens after the upload of the first frame, and so on. This isn't trivial to implement as I'd have to first adjust the hwdec code to use asynchronous cuda; without that, there's no way to use the semaphore for synchronisation. This should result in fewer intermediate buffers being required.
2018-09-30 01:00:19 +00:00
}
hwdec_cuda: Use the non-deprecated CUDA-GL interop API The nvidia examples use the old (as in CUDA 3.x) interop API which is deprecated, and I think not even functional on recent versions of CUDA for windows. As I was following the examples, I used this old API. So, let's update to the new API, and hopefully, it'll start working on windows too.
2016-09-10 03:07:43 +00:00
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
error:
vo_gpu: hwdec: Use ffnvcodec to load CUDA symbols The CUDA dynamic loader was broken out of ffmpeg into its own repo and package. This gives us an opportunity to re-use it in mpv and remove our custom loader logic.
2018-04-15 16:06:34 +00:00
eret = CHECK_CU(cu->cuCtxPopCurrent(&dummy));
hwdec/opengl: Add support for CUDA and cuvid/NvDecode Nvidia's "NvDecode" API (up until recently called "cuvid" is a cross platform, but nvidia proprietary API that exposes their hardware video decoding capabilities. It is analogous to their DXVA or VDPAU support on Windows or Linux but without using platform specific API calls. As a rule, you'd rather use DXVA or VDPAU as these are more mature and well supported APIs, but on Linux, VDPAU is falling behind the hardware capabilities, and there's no sign that nvidia are making the investments to update it. Most concretely, this means that there is no VP8/9 or HEVC Main10 support in VDPAU. On the other hand, NvDecode does export vp8/9 and partial support for HEVC Main10 (more on that below). ffmpeg already has support in the form of the "cuvid" family of decoders. Due to the design of the API, it is best exposed as a full decoder rather than an hwaccel. As such, there are decoders like h264_cuvid, hevc_cuvid, etc. These decoders support two output paths today - in both cases, NV12 frames are returned, either in CUDA device memory or regular system memory. In the case of the system memory path, the decoders can be used as-is in mpv today with a command line like: mpv --vd=lavc:h264_cuvid foobar.mp4 Doing this will take advantage of hardware decoding, but the cost of the memcpy to system memory adds up, especially for high resolution video (4K etc). To avoid that, we need an hwdec that takes advantage of CUDA's OpenGL interop to copy from device memory into OpenGL textures. That is what this change implements. The process is relatively simple as only basic device context aquisition needs to be done by us - the CUDA buffer pool is managed by the decoder - thankfully. The hwdec looks a bit like the vdpau interop one - the hwdec maintains a single set of plane textures and each output frame is repeatedly mapped into these textures to pass on. The frames are always in NV12 format, at least until 10bit output supports emerges. The only slightly interesting part of the copying process is that CUDA works by associating PBOs, so we need to define these for each of the textures. TODO Items: * I need to add a download_image function for screenshots. This would do the same copy to system memory that the decoder's system memory output does. * There are items to investigate on the ffmpeg side. There appears to be a problem with timestamps for some content. Final note: I mentioned HEVC Main10. While there is no 10bit output support, NvDecode can return dithered 8bit NV12 so you can take advantage of the hardware acceleration. This particular mode requires compiling ffmpeg with a modified header (or possibly the CUDA 8 RC) and is not upstream in ffmpeg yet. Usage: You will need to specify vo=opengl and hwdec=cuda. Note that hwdec=auto will probably not work as it will try to use vdpau first. mpv --hwdec=cuda --vo=opengl foobar.mp4 If you want to use filters that require frames in system memory, just use the decoder directly without the hwdec, as documented above.
2016-09-04 22:23:55 +00:00
if (eret < 0)
return eret;
return ret;
}
vo_opengl: separate hwdec context and mapping, port it to use ra This does two separate rather intrusive things: 1. Make the hwdec context (which does initialization, provides the device to the decoder, and other basic state) and frame mapping (getting textures from a mp_image) separate. This is more flexible, and you could map multiple images at once. It will help removing some hwdec special-casing from video.c. 2. Switch all hwdec API use to ra. Of course all code is still GL specific, but in theory it would be possible to support other backends. The most important change is that the hwdec interop returns ra objects, instead of anything GL specific. This removes the last dependency on GL-specific header files from video.c. I'm mixing these separate changes because both requires essentially rewriting all the glue code, so better do them at once. For the same reason, this change isn't done incrementally. hwdec_ios.m is untested, since I can't test it. Apart from superficial mistakes, this also requires dealing with Apple's texture format fuckups: they force you to use GL_LUMINANCE[_ALPHA] instead of GL_RED and GL_RG. We also need to report the correct format via ra_tex to the renderer, which is done by find_la_variant(). It's unknown whether this works correctly. hwdec_rpi.c as well as vo_rpi.c are still broken. (I need to pull my RPI out of a dusty pile of devices and cables, so, later.)
2017-08-10 15:48:33 +00:00
const struct ra_hwdec_driver ra_hwdec_cuda = {
vd_lavc: add support for nvdec hwaccel See manpage additions. (In ffmpeg-mpv and Libav, this is still called "cuvid". Libav won't work yet, because it has no frame params support yet, but this could get fixed soon.)
2017-10-28 17:59:08 +00:00
.name = "cuda-nvdec",
.imgfmts = {IMGFMT_CUDA, 0},
.priv_size = sizeof(struct priv_owner),
.init = cuda_init,
.uninit = cuda_uninit,
.mapper = &(const struct ra_hwdec_mapper_driver){
.priv_size = sizeof(struct priv),
.init = mapper_init,
.uninit = mapper_uninit,
.map = mapper_map,
.unmap = mapper_unmap,
},
};
Reference in New Issue Copy Permalink