Part of it is absolutely race related. If the president's grandfather were Irish, would people be frothing at the mouth and yelling for him to go back to Ireland? I don't think so.

Some people do have legitimate arguments with his policies, due to their own political beliefs, and I'm not talking about those people, but rather the ones who find fault with every single thing.

http://www.huffingtonpost.com/2013/10/01/jimmy-kimmel-obamacare-prank_n_4022424.html is a good example of how labeling anything with "obama" will pull the crazies out and set them on a rabid rant.

If you've never seen The Daily Show, you should look up a few recent episodes and watch. :)