Well my first advice here is that you are going about things wrong.
Unfortunately this is the same thing everyone who is new to car audio does when they want more bass. The truth is anytime you boost a frequency (turning treb up, turning bass up, turning bassboost up. upping your EQ) Are all things that are going to add distortion to your signal. Distortion will cause clipping, distortion and clipping together will destroy your speakers/subwoofer. The unfortunate truth may be that your system doesn't get as loud as you would like, and the solution to this should be to get different equipment.
So I want you to put the RCAs on the Sub/W. Turn bassboost completely off. Turn LPF down to around 80hz. Use a DMM to set your gain by the equation √(total wattsxtotal ohms) IE you have 2 dual 4 voice coil subs that are rated 150rms, you have them wired to 1ohm at the amplifier. Disconnect your sub(s), turn gain to 0 turn volume on HU to 75% of max. play a 50hz tone either via CD or aux. Hook up Digital Multimeter (20$ at radioshack) to output channel on amplifier set DMM to AC voltage. Now turn gain up slowly until you reach the number you need from the equation. So 300watts x 1ohm = 300, √300=17.32 so turn up gain until DMM reads 17.32 for this example. When you have that all set, that is now your max volume on your HU. Turn any bass and bass boost to 0. This is as loud as you can get safely for your subwoofer(s). As for your speakers they may need to have a HPF on them so they don't distort at this volume, or you may need to set HU volume to the max that the speakers will take without distortion, then run the testtone and set gain.