This is making one false assumption... that there IS NO CLIPPING.
99.9% of the time, having too small of an amp == clipping, a LOT of the time.
Many amps have very POOR clipping behaviour... it's not unusual for them to momentarily "latch up"... and send out bursts of DC on the output. Doesn't take much DC, to make a voice coil light up like a space heater element. Without the cooling effect of voice coil movement in the gap (which a small amp CAN'T generate, since it can't generate enough AC voltage swing), the DC will readily fry the adhesives the speaker voice coil is assembled with...
I can readily demonstrate the effects of hard clipping of a small amp, on a big speaker... I work on PA equipment, for a daily gig. Lemme hook up a 150w/ch amp- something, maybe, like a Behringer (ie, bulls***) amp- to a 1000w Electrovoice EVX180 18" pro subwoofer... wind the volume up to max, and watch the amp go into momentary DC. Bet the woofer will die within 10 minutes...
The point here: CLIPPING can destroy a speaker, NO MATTER if the speaker is rated at MUCH more power than the amp is rated...
Regards,
Gordon.