I think Merlin is right on this one along with the OP. To be OK you have to be pretty serious. I would guess that those who are discounting the level of dedication either are ignoring the amount of work they've put in or are taking for granted their natural talent for racing, which a lot of us don't have.
The comparison to golf is just inaccurate. When have you vomited from the effort of training/playing golf? Having played golf, I just don't buy that 11 hour weeks of training/playing golf are as taxing as 11 weeks of training for racing on a bike. Having said all that, maybe the difference in effort isn't the point.
As taxing? No, not at all, but as demanding or challenging to be good, yes, 100%. I presented golf as an example of a non-cycling activity that requires extensive dedication to be "good".
Also, yes, you have to put in time and effort to be good at racing, but that requirement isn't limited to cycling, but is a common theme among pretty much any competitive endeavor.