The change you hear after adding fill has little to do with the driver appearing to be in a larger box (this does occur but the audible affects of this are very subtle) and everything to do with the damping properties of fill.
It's similar to how shock absorbers function on a car, if your shocks are worn out the car keeps bouncing for a little bit after it hits a bump, this is what can happen in a sealed box with no fill, adding fill is like fitting a new set of shocks, it makes the sub stop faster after a bass hit instead of ringing.
The end result of this is it lets the sub more accurately track the audio signal, fast and complex bass passages will sound fast and complex, rather than a smeared boomy mess.
Note that some subs and enclosure types have better inherent damping than others and in some cases the effect of fill isn't very noticable.