Make XAN decoder return meaningful error codes.
Originally committed as revision 25516 to svn://svn.ffmpeg.org/ffmpeg/trunk