I would really not recommend cutting deep into MDF with a 1/4" bit. I have had many bits snap on me, and at 20-30k rpm they go FLYING. It is not safe to cut with such a small bit regardless of speed imo. If you want to be safe, don't cut deeper than the bit is wide. It will only take ~30 seconds more to make 2 quick shallow passes.
Like you said, If your circle is off then your guide pin is too loose, or the router base is not very solid. Possibly when you are plunging you may be tweaking or rocking the router a bit. Make sure everything is nice and tight. I have no difference in the location or diameter of the cut when cutting in steps.