That's still service, guy. Now, if they're letting your dirty plates pile up in front of you, different story, but if they're constantly clearing your eating area and refilling your glasses, I say that's worth at least 10% or $1-2 for buffets < $10. But always check the bill, as some places tack on the gratuity regardless of service.
If the person is just refilling your drink and taking away dishes isn't really service. Service is personalizing with the customer, and if you can't do that then don't expect a tip.