Part of it is that it's not very interesting to watch if you don't know what's going on.
Another part of it is just the trend over the years. Starting sometime in the middle of the century, bicycles became a popular way for kids to get around. People see bikes as a hobby for kids, and transportation for poor people. The idea of adults racing on them becomes ridiculous given this mentality among the general public. Even my parents think I'm weird.
It will take many years of good performances internationally from Americans and American teams before it starts to become a legitimate sport. But this is unlikely to happen when the most promising children are playing baseball and football instead of learning to race. We have to get the kids racing before it will eventually become a legitimate sport in this country.