Take a look at the properties of a subwoofer, every material has slightly different properties at one temperature from the other.
A rubber surround will be more supple and pliable in the heat than at frigid temperatures. Foam, due to the large amount of air bubbles, won't change much. A woven material like spiders are made of is also quite impervious to drastic mechanical changes due to temperature.
One thing to consider however is the glue that holds the driver together. The glue keeping the cone on the voice coil, the glue on the dustcap, the surround, etc. Glue holding strength can vary greatly with temperature, and you run the risk of tearing "brittle" glue joints in some cases.
And other than your amp, which BassAce mentioned, just take it easy if it's seriously cold. I'm talking 20 or less. If you let it warm up a few minutes then I see no problems.