You should caveat that with "audible distortion." Every amp has distortion, simple fact of life. Class D amps have more than Class A or A/B. When you drop the impedance, distortion increases. The important consideration is when does this distortion become audible.
With a poorly designed or built pre-amp stage in an amp, turning the gain up usually increases noise, not distortion. With a good amp, neither is a factor regardless of gain setting. When you increase the gain to the point of inducing clipping, however, distortion goes through the roof.
Unless you have a golden ear and some really choice speakers you usually can't hear distortion below 1% except on pure tones and then only in an A/B comparison. With music, the threshold is a good bit higher than that.
The only time I would choose 4 ohm over 1 for wiring subs is if I could get the same power (or at least more than adequate power) from the higher impedance and even then it wouldn't be because of distortion, it would be because of amp efficiency.