if you go too big on ether the enclosure volume or port area you will increase the chances of the sub to bottom out or get over powered even with less than the rms rating of the sub. proper enclosure design is the key word. you want the port to be big enogh to not have port noise at the highes power level that the sub can handle. any bigger then that is just a waste.
yeah you need to find a happy medium. if the port is too big the sub will act almost as if it were in a free air environment and could damage it depending on the power given to it.