It is a completely semantical argument, where you are setting one reference point and they are setting another. Both arguments are completely plausible.
From a subjective standpoint with subwoofers, flat axial response is one of the driving perceptual factors, which is especially unfortunate in a vehicle where most listeners are incapable of discriminating between the room effects, the driver, and the enclosure.
It is true that we don't listen to BL curves. The Klippel, for example, does statistical fit analysis of contributors to distortion. The only way to know how much distortion is being generated is to physically measure it. However, we know that there is a strong correlation between the linearity of BL, Cms, Le and the physical distortion that is created from a lack of linearity. From an engineering standpoint, it is easiest to start by targeting linearity.
Can you hear the distortion? Well, it depends on the spectrum and amplitude of the distortion, the stimulus, etc. However, I would approach all systems (particularly from the speaker aspect) the same: the elimination of distortion should be a primary goal since the goal of any audio system is to reproduce the source, rather than produce something different that we may interpret favourably.