do you have a brace going from the baffle to the back of the box?
the back of the box is also a huge piece that will flex easily. I would recommend two wooden dowels going from front to back.
this is one of my old boxes:
it is no where near the size of yours and I was having mad flex problems before putting a brace from baffle to the back. Mine wasnt double baffled, but i would still recommend extra bracing. This was with about 1500rms going to this sub.