i'm not too careful with mine. Can't pull it if you're too gentle. Sometimes you gotta take a leap of faith and pull it like a mofo and it'll just rip right out.
Now when I need to take out my hu, I rip it out like velcro tape.
I remove my ashtray and pull the bottom of the lip towards me hard. There are little clips inside that hold it in place. It's due to those clips that I rip it out, else the latches will bend and break.