avcodec/libvpxenc: Check for av_malloc() failure
Fixes CID1271045
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>