Sure it matters. you dont want the sub too close to the port because the air is going to take the path of least resistance which would be out of the port opening.
It kills me when i see someone build a huge box but then put the speaker right next to the port. how does the remaining air to the...