The only way to damage a speaker is to overpower it (whether that is thermally or mechanically); clipped signals have a lower crest factor than a sine wave so there is more power under the curve. A clipped signal alone will not damage a speaker at all; the only damage can occur from sending more power than the speaker is able to dissipate, which can be achieved from music, sine, or square waves.
Clipping is bad for achieving the most accurate reproduction of the input signal, but it not necessarily damaging.