h264: move resolution change check further up.
This prevents some variables from being changed in case of a rejected resolution change.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind Signed-off-by: Michael Niedermayer <michaelni@gmx.at>