Not sure what causes this, but I've noticed it with mine as well. Usually on a page refresh, I'll see it change by one or two votes.
@pembo210 and @polsaker have been mum on the issue. I smell fuckery! /s
I can't tell you the cause. Something cache related.
But they sync up after a short time.
I always knew @Polsaker secretly fiddled with the scores!