No, No, No, No.Let me reiterate that ~
NO.
If silicone sealant was detrimental to rubber speaker surrounds (I cannot speak to its effects on foam surrounds) then doing this would have been a really bad idea:
See the area enclosed by the red line. The idiot that I had helping me put these 10s in their box made a slip-up and while trying to sink the mounting screws screwed up and ended up punching a very small hole in the surround of that sub. I coated that section of the surround with a thin layer of
silicone sealant to make sure that very small hole did not become a larger one.
That was about three years ago now.
No adverse effects as of yet so I would assume there aren't going to be any.