How noticeable is it? I guess it depends on the extent of the damage... some speakers won't play if they get too hot and that heat causes severe damage.
Kind of like a car I guess... if the A/C breaks, you can still drive it but it's not as much fun. But if your car breaks down and you have to hoof it, you are really going to hate your car.
Clipping does not matter... it's how hot your speaker/coils get... you can heat up a speaker, clipping or not. But I will say you are more likely to get it hotter with clipping. But it is not the end-all be-all of speaker damage.
If I hold a lighter under your boo-boo for three minutes (moderate power clipping over time), or hit you with a flamethrower (mondo musical peak), either way, it's still going to leave a mark. Would you like to be burnt to a crisp quick or slow-cooked?! Do you smell what I'm cooking?! //content.invisioncic.com/y282845/emoticons/flamer.gif.1fb8372af2de4b576aa9428ccc8a8f2f.gif