At first I thought maybe I was indeed feeding it too much power. I had it running off an Alpine MRV-1507 bridged with very modest gain settings. That amp is supposed to do 600@12v / 900@14.4v. I decided to step down the power to see if it would help to an Alpine MRV-T707 which is supposed to be 240@12v / 340@14.4, and that didn't solve the problem.
I am going to try and find some time to remove the speaker from the box and follow these guys suggestions and free air it to see if I can notice anything out of the ordinary, i.e. wiring actually hitting the cone, tinsel leads, or pretty much any other condition that might be "out of the ordinary".
Also, I have mine in a sealed enclosure. It has been years since I constructed it but I know I built it to Kicker recommendations for sealed so I'm thinking 1.25/1.5 ft^3