I once got placed dead last in a pleasure class because the judge didn't like the colour of my appy (varnish roan) a friend of mine who is a judge was watching the class said after that the horse that won (blood bay) started bucking in front of the judge.
I think the biggest thing to remember when you are riding a coloured Sporthorse is that you are going to stand out, good or bad. As you put it, your eye is drawn to the colour whether you like it or not. I show Hunters and had an Appaloosa previously and now a Pinto Canadian Sport Horse. I wouldn't trade either of them for all the bay's and chestnuts with Chrome in the world!
I have a white few spot Appaloosa mare and a black and white pinto mare and when I have shown them in hunters and dressage I have definitely noticed a bias under some judges (favorable and unfavorable). My white mare is already an established event Horse and my pinto is just moving up the levels to get to that point So it doesn't bother me, but yes, definitely notice it.