If anything, you're just losing out a bit more on the SPL portion. I doubt you'll hear any significant SQ differances in either sub in a well built box.
If you go with 12s, you theoretically have the potential to get about 1.5 dBs louder because of the increased cone area. Sound quality differences should be undetectable, as long as the boxes are both properly designed and built.