If it makes you feel any better, I don't think you're wrong. I used to think the same way, but I got into baseball again this season. I played a bit this summer, too, which probably helps, and going to the game and watching it does beat the shit out of watching it on TV, as well. Pretty much anything is better when you're in the VIP section of an area and being waited on hand and foot.
As for race car driving, I think it's just used as an excuse for drinking. "500 laps. Well, I can't get through that without some more beer!"
And, judging by some of the crashes, that quote could easily be a spectator or a driver.