I say this in a nice way, but maybe you're a jagoff. Maybe it is the way in which you interact with people that gives them that vibe off of you. It doesn't matter if you do or don't get smoked out with the dudes around the block every night. I never touch the shit, but I can be around my friends who do without making them feel like I'm better than them. Maybe it is you. Maybe it is you correcting your buddies and looking down on them and others because you are more privileged. I grew up poor, but at the same time, I went to the best schools in my area. Only time I stepped in public schools was for my buddy's graduations. Chicago Public school education has to be one of the worst in the world. But I have friends who made it through and a few are doing better than me, and a lot more aren't. But we never look down on the ones that aren't because of it. It is the life they chose.
So watch the way in which you interact with others rather than the way you speak. When I go to other cities, their diction sounds a lot different than mine. I go to Ohio and they sound "white" I guess. But those same Ohio cats with their sounding out every letter doesn't make me think they are better than me because I don't. Never dumb yourself down, but try not to Big Brain Brett cause you got you some schooling.