Oh relax. It's not the end of the world if someone disagrees with you and gives you a couple of rate downs. Ignoring it might not make you feel good, but it will usually make you look good.
Sorry Joz, if I could do something about the random rate downs I would've by now. All you can do is just ignore the troll and remember that they're a coward. Works for me, sorta.