avcodec/dnxhdenc: check that qmax is within the supported range
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>