avcodec/dpx: return different error code for unsupported depths
Also give better message in such cases.
Signed-off-by: Paul B Mahol <onemda@gmail.com>