You sure that center part is supposed to come out?
It looks kinda similar to the ball joints that I replaced in my dad's truck. The entire thing (bolt thing, rubber grommit, and some metal around the outter edge) had to be pressed out. I think Autozone had the tool to do it.