I've heard innocent people called a "racist!" indirectly & directly so many times that the label has lost all credibility with me.
It's the fascists who call other people "racists!" that are the real problem IMHO.
So have I, and in fact, I ended up in a thread much like this one on a left-oriented forum I used to post on because I stood up against the racist label that they were using unfairly. I argued that calling someone a racist when they just disagree about a policy is as bad as using racist slurs yourself.