I read this sentiment fairly regularly on the boards here but I've never seen it out while actually riding my bicycle. I suspect it's an attempt to rationalize why others don't take our passion as seriously as we do, but I just don't see the evidence that "most" or even very many people feel that bicycles (or motorcycles) are "toys".

Since this kind of behavior happens to motorcycles, classic cars, and in my own personal example, riding a horse into town, the reality is that people just get excited when they experience something novel, exciting, unique, or nostalgic.

In the US culture we have attached significant value to modes of transportation (usually automobiles) and especially private property. So while I understand that people will develop strong feelings and reactions to things that seem to violate their personal space it's probably worth remembering that it's a fairly unique and modern concept to conflate "private" property with personal boundaries. At the end of the day, especially considering what one will reflect upon when getting ready to travel beyond this reality, it's probably not worth stressing over. And it's definitely not reasonable to compare fiddling with an object to fondling another person (and a minority at that).