My box was right in the recommended enclosure specs, so was not extra big.
Well guess I just had shitty luck then, I didn't set anything wrong. And if I did both subs would have ended up with the same problem because both subs are in the same single chamber box and they're both on the same mono...