I have read on a different forum that in order for a shop to sell Specialized bikes, they have to stock a large range of their bikes, parts and accessories. This isn't great especially for the smaller shops as floor space is limited, and products in such quantities are difficult to sell. Small shops tend to have unfavorable opinions about them.
As far as the bikes themselves go, I know that until this year's Diverge models, they were essentially cheating shorter/smaller riders with their geometries (Sheldon Brown has a great article on how companies take engineering short-cuts and Specialized took them aplenty). Stack-reach ratios on the smaller frames were less than 1.4 while those on the larger ones were above 1.6, making them not really an 'endurance fit' for the vertically challenged. They seem to have gotten their act together on this year's geometries though.