Make sure that the hole is drilled perpendicular to the wood and it will minimize the problem. Don't use anything other than the smooth ground pivot pin that came with the jig either. It won't mess up the holes in the jig. Also just let the tool do the work and don't put much if any pressure on it.
The setup is everything. Use a sacrificial board behind the work and set the cut depth to about 1/8-1/16" into the backing board. Drill the center hole into the backer board so the center point can't slip laterally. I hold the whole thing together with two sided tape. It keeps the rings from spinning once you've cut all the way through and keeps the ring from being pulled into the bit once it's free. Cut the outside first and then the inside.
The most important thing in my opinion is using the correct bit. A straight cut bit isn't it. It places all the cutting load horizontally causing a lot of stress and deflection of the bit. It also leaves the sawdust in the slot that it cuts causing more binding and greater tool loads. Trying to cut thin walls with a straight cut bit can be really frustrating because the side load on the bit will break the ring. I use a spiral up-cut bit. It loads vertically, pulling the router into the work. It also clears the dust out of the cut and keeps it from loading up. The only caution with this type bit is that you have to make sure that the collet is really tight. If it isn't the bit will pull loose from the tool and into the work. Eventually it will cut all the way through everything an into your work surface. The wings on my saw table have a few scars from this happening with the worn collet on the Skil router that I mentioned above.
Ramos-I've had this particular Skil unit for about 3 years. The collet is part of the motor spindle and is total garbage, IMO. If I'd known about it when I bought it, I wouldn't have.