Welcome to the world of logarithmic scale.
The issue here is that sound intensity revolves around the foundation that 1dB is about the audible threshold for humans hearing a difference, but 3dB requires doubling power to achieve, so your average source unit is rolling somewhere around 1 to 1.5dB per "click" on the volume knob. If yours jumps from 4-6V I'd figure you're about 1.3dB per click. Generally speaking if the numbers on your source unit go from 1-30 it'll be 1.5dB per click and a 1-50 will be right around 1dB per click. 35 and 40 total clicks in your volume sweep put you somewhere around 1.2 to 1.3.
Long story short, this is how it's supposed to work and this is why you can go from playing music just fine to hard clipping and/or breaking **** really quick with just 1 more click of the volume knob once you've hit limits.
The big problem with all of this, in gain setting, is that if you throw in a music track that's mixed a little lower you may still need 2-4 more clicks on the volume knob to really get the output you want and get your source unit up to 4V or whatever it is you're trying to match to your amp. At the end of the day, you need to plan around the wider range of music to which you listen, gain accordingly, and just be mindful with that volume knob when the louder songs come up through your playlist.
Those Kenwood units should also have sub gain functions in the "sound settings" menu and also gain functions for each source which may help you dial this all together to match to your liking.