Unless you've got a 20v signal I don't see a problem. Even then "problem" would be subjective.
A 20v signal with the gain set at 4v would be like the common newb running the gain peaked with a standard 2v preamp. You just run into clipping before full volume. It's not like the input stage will burst into flames.