Wrong, the reason why you hear any difference is because you have now shrunk the volume of the "enclosure" the driver is playing in.
In doing house remodels, I've done tests of walls with insulation and without. The difference is minimal as to how much sound can be reverberated to the other side. It's done in the ceilings of houses to avoid heat/cold transfer thus allowing your home to stay at the desired temperature more easily.
Using home insulation is no different than using poly fill, except for the fact that one can cause itchiness, molds easily and over time deteriorates with various temperature changes, age and weather. The other loses no volume due to deterioration and has no properties that'll cause histamine reactions nor does it mold. Tbh the home insulation idea is just about the same as people presenting peel and seal as a sound deadener.