That would be interesting. I usually know who my fairies are, but sometimes it's a mystery.
I think it could still be exploited by using alts, but far less people are willing to go to that extent.
I personally would have no problem making my votes public, but I could understand if some others didn't.
I don't actually mind them. It's validating sometimes, and doesn't actually hurt anything.
As a bonus, I know of an account that started giving out upvotes, only because they couldn't downvote me anymore. I'll take one for the team :)
They only did it at Poal because many were complaining about 'brigades' as often happens. That gave them the ability to write up the down vote brigades for the admins. Me, I could care less ... if I don't like something, no vote at all is my go to. After all, they're only fake internet points anyway, though ... so only a few who worry about things care.
Poal started a deal where you can check who down voted. I'm not sure if that's a good idea or not, but they were being plagued by goats coming in just to harass them. It stopped that quick.