At high volume I cant even hear the speakers because of the bass, but they do distort at high without the subs but with the factory bass...
Your subs are drowning out everything else. Is that the goal you wanted (very high subbass SQL/SPL)?
If not, then your HU and speakers are not providing enough SQL/SPL to keep up with your subs. You'll need to find balance in your current setup if you are not planning to run a really decent set of front speakers and an amp to close the gap. This will involve lots of tuning and tinkering.
1. A likely place to start is to turn the sub gain down so that your sub stage will not continue to overpower the rest of your system.
2. Find a volume setting on your HU (flatten all the EQ first) that doesn't provide distortion. This will be your maximum capable clean listening volume.
3. Now slowly work the sub gain up until you reach a nice balance between subbass sounds and everything above that.
Remember, a well tuned/balanced and imaged system can only get as loud and clear as the weakest component will permit. Because your HU and speakers are relatively weak (by the sound of your descriptions) in comparison to your sub stage, the subbass is murdering everything, and that's not something bass blockers will fix.
Let me try to paint a visual by what I mean here:
SPEAKERS
powered off HU (20wpc rms?) can only get to 30/50 volume before they distort.
...not much SQL/SPL
SUBS
clean install and powered by a capable amp. Gain is set to provide max unclipped signal.
...loads of SQL/SPL, even at 30/50 HU volume.
RESULTS
high powered sub output is eating low powered speaker output for breakfast.