avcodec/aacenc: check input for NaN
Fixes Ticket3762
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>