Unless you play with the vehicle not running for extended periods of time, you don't need a battery isolator. And while it is best not to commingle battery types, one battery will no kill the next. What will happen is that the wet cell will rob the float charge from the agm(s). All that means it that the resting voltage of the agms will fall a few tenths of a volt after sitting for a long while.
As for battery capacity, you have 110AH for a ~4000 watt amp. You need about 55AH per 1kW so yes, you need a second battery for the amp. Since your front battery is probably only a 50ish AH battery, don't include it in the equation, even though you should use it.
But in the end, it's all up to you. You have a powerful, expensive amplifier in your car. Power is the fundamental key to both good amplifier performance and a long, safe lifespan of the amp(s) so consider that. I prefer to overbuild my power capacity so that my amps perform well and aren't in danger of dying themselves or, cooking my coils.
