Without seeing the inside of the box, nobody could possibly tell you if the port is too small or too big. Within the constraints of the purchaser(you) I have a feeling the box is the perfect box to meet your needs. Now if you had 12's, box would be different, 18's, then that would be a different install. My only concern is the amount of wires being ran directly off a single battery terminal. But that is just personal preference, I prefer to use fused distribution blocks for the equipment that uses less power. Just looks cleaner to me.
Edit, I didn't realize you put up the port dimensions, and yes, they are a little small. But within your constraints, they are needed. You're losing output, but until some of your variables change, the box will have to stay the same.