I'm an audio mastering engineer in Colorado where we often experience temps in the teens below zero (F). During the audio mastering process we play a test CD in a car stereo in addition to testing on a variety of home stereo receivers and speakers. What we have found is that temperature can drastically affect bass performance so we make it a point to never test in a car when the outdoor temps are below about 30 degrees F.
Those of you that live in similar environments have probably noticed that the seat cushions of your vehicle become quite stiff when temps drop into the single digits or colder. Just as the polymers in your seat cushions stiffen in cold, so do the materials used to manufacture the surrounds in audio speakers. The "surround" is that flexible ring that connects the speaker cone to the speaker basket. In one of our tests perfomed at 10 degrees F (w/o car heater), bass was reduced by 24 dB compared to tests done at 70 degrees F. Some speakers are more affected by cold than others depending on the speaker materials. Speaker surrounds can be made of butyl rubber, Santoprene, and several other polymers. Each material has its own characteristics when exposed to cold and therefore affects speaker performance to varying amounts. If the temperatures are cold enough its entirely possible that the materials will become brittle and break. I would expect that you don't have to worry about breakage if the speaker surround is made of butyl rubber as this same type of rubber is used in automobile tires.
In addition to the speaker surround, there are also speakers with "liquid cooled" voice coils. Oil is the liquid used in this application to prevent freezing, however even though the oil doesn't freeze in typical automobile applications, it does thicken and affects movement of the coil in the magnetic gap which in turn affects audio performance. To address the issue of damaging your amplifier, I would imagine that as the surround becomes stiff that the load on the amp in increased as is the chance of damaging the amp.