the biggest thing would be vibrations and heat dispersion...
if the magnet has a pole vent on it, then taht needs to be clear to allow voice coil cooling... if you can pad the wall so taht it won't vibrate (I am doing this with my fibreglass - lousy K... **** wide basket...) then it shouldn't cause too much problem on that front... cooling is the biggest thing..