A line driver doesn't magically make your amp play louder.
An amp can only physically generate a certain amount of power. It takes a low voltage signal and amplifies the signal, that's it. Just because the input voltage is greater doesn't mean somehow the amp can step the voltage up to a level past where it was before. If you add a line driver and your system plays louder without distortion that just means that you didn't have your gain turned up enough without the line driver.
The only time a line driver will do anything useful is when your source voltage is so low that your amp can not amplify it enough to reach it's maximum output without clipping.