OK. I'll reword it. You can't say ""that wasn't a violent action, because my punch isn't very strong" after punching someone in the face.
If you treat someone in a way that negatively impacts them and do so simply based on their race (skin color, ethnicity, etc), the you can't say it is not a racist act or that you are not racist.
By definition, doing racist things means you are a racist.
It is not necessary for you to treat every single person that way. If you do it to one, it is racism and you are racist.