Ideally you would get best results if you could run an oscilloscope off of the speaker terminals on the amp, what I would do for just a REAL simple test is to measure the voltage coming off of your RCA leads with AC current on the DMM, preferably one with without minMax feature so that you can see the rise and fall and get a pretty good idea of peak voltages, then hook them up, play the same test tone and adjust your gain until the voltage stops raising and just levels off, turn it back down to where it hits that peak voltage only where the RCAs were hitting a peak voltage.