I can See both of your points and some is great to note. But confusion is clear here.
The question of the op is about what to do when he inverts the drivers and what affect that will give on the systems performance. It has been stated, and I will state it as well. It is a good idea to reverse polarity of the drivers when inverting them as long as they are in phase with the system, not themselves. Wenn is right about timing effects and phase being an issue. The reason they are, is that even though a 180 degree switch is given in ac current to complete a full cycle of a some wave, the time it arrives at a specific listening position is critical when dealing not only with crossover frequencies, but the enclosures acoustical changes, such as the common known resonances that occur. These are merely multiples of a harmonic, so they can extend and affect the output of a signal well into the 8th degree if the response is capable of such a curve from the amplifier and system setup. So, polarity is important, but not to be confused with phase. The actual phase of the system will differ from that of a single part of the system, such as the subsystem. This is where polarity is localized to the drivers when they act as a single source of that frequency range. When they act as a whole full range setup, then phase is important more so than polarity as polarity should be figured before combining the system components. In other words, you want to make sure the polarity of each part of the system is done individually before playing them together. Then, the phase is given at specific listening positions and can be more related to direction and distance. So, that being said, do not confuse polarity with phase. They are two different parts of a systems functionality. the changes that occur with what has been stated so far from you guys, is due to physical changes, not electrical. Those physical changes are from enclosure performance. This is where wenn may be confused, but on the right track. Yes, you can notice a change when reversing polarity, but only if the enclosure allows that audible change to occur. If the driver displacement has a large enough volume, and the ratio of that volume greatly affects the volume of the enclosure, then audible changes will occur. So, its really application dependant. But does it have an effect on performance? At higher volumes, not as much. Changing anything about a systems setup will have an effect on everything, but it is the amount of change that is important. In this case, its not major. If having the subs normal mounted and they were in phase with the front stage (i.e. Within 90 degrees ±45) then that is the way it should be wired. Output will only be noticable at the position where in phase correlation occurs. So, if you invert them, wiring should change to keep phase relative. The 180 degree switch will inherently change the way it arrives at the listening position. 180 is not 0. 360 is 0. 180 will have a reverse effect on the sound at the listening point and any resonances that were excited, will be affected, hence lower or higher audible output, again, if the design allows it to be audible. So, experimentation is key in this and should be al2ays tested above any calculations to find what will work the best in your specific setup.
So, if inverting, and it was in phase before, reverse polarity to keep it as close as possible to a system in phase setup as possible. It can easily be explained as four drivers of the same, being utilized for two different purposes, two for each purpose, but you can still wire them in and out of phase. First as a subgroup the bass driver need to be in phase, and the mids need to be in phase, then as a group, they need to be in phase. Changing the polarity of a subgroup will still cause unnatural effects to occur, just as it would in a stereo setup with two driver. And polarity changes when inverting is done to the drivers.