It's somewhere in between for me. I mean sure it can be annoying sometimes, especially when it seems their intent is more to incite than inspire. Sean Penn comes to mind. But just because they're famous actors or athletes doesn't mean they're dumb and incapable of having sound opinions. That's unfortunately a common assumption. And I like when they poke fun at it, like Alec Baldwin does on 30 Rock.
I don't really let it impact my opinion of their talents. I'd like to think I'm smart enough to listen to what they have to say (should I so choose) and form my opinion regardless of whether I think they have a nice beat to go with it.
All that said, this is one of the greatest SNL skits ever: