Here's a download link to a -5 db 50hz test tone.
https://www.dropbox.com/s/ln6ffxqd5e3xtm4/Negative%205%20db%2050hz%20test%20tone.mp3?dl=0
I saw that same amp push two hdc3 12s hard easily on an 180 amp singer alt and single XS power D3400. So no its not the amp, something else is definitely wrong with your current setup. Basically if you get any amp over 1k this issue will continue to happen Reason why i say this is because you said THERE IS NO DIFFERENCE between 2 ohm and .5 ohm which indicates your electrical is no where near adequate to provide proper power for this amp. When it is ready, you will notice a massive difference from the wiring switch alone.
I had a buddy with a stock corolla with 80 amp alt and everything stock try to run a power hungry hifoncs brx 2400. Result is, the output was weaker than his kenwood 500 watt rms amp. Tried lowing impedance from 4 ohms to 1 ohm. No difference but the power got dirty and clipped hella fast on either ohm load. On the multi-meter, Vac did not get past 20 volts thats only 400 ish watts . That same amp slammed on my system with my 320 amp alt and battery bank and did not get dirty. Vac read 48 easily on the multi-meter. Lack of actual current causes major clipping and low power output thats a guaranteed fact.
So does your boat have an alternator? how many amps does it provide? How are your grounds done? Did you do your voltage test at the amp power and ground terminal or at the battery?
Also any pics of the inside of the wire? I would like to see how thick the wire actually is not including the outer jacket.
Another point i'd like to make, if you have the sub in a sealed box. Your subsonic filter should be off. Put it to the complete minimum you can get it to.