Explicitly forbid using a non-RGB colorspace with RGB pixel format or vice versa. This mirrors identical logic from vf_scale.