You want the damping factor to be above a certain value. But going beyond that yields nothing beneficial. DF isn't a spec where higher is always better. Just as with slew rate, there comes a point in which going any higher doesn't yield any audible difference. This point is typically considered to be 20. So if the amp has a DF of >20, then it has adequate damping and there will be no audible effects.
Really, the only time damping factor would be a remote consideration is for a tube amplifier, which have very high (comparatively) output impedance. Most any solid state amp will have more than adequate enough damping.