Because you probably have not met any fit riders on recumbents. Many recumbent riders are older or larger riders who want to keep or start riding longer distances without the normal discomfort associated with uprights. These riders are usually less fit and less interested in going fast.

However if you had come across a ft rider interested in recumbents because of the speed potential then you would have likely seen very different bikes and riders. I am only moderately fit and overweight but I can do 18 to 20 miles in a one hour road ride including modest hills and obeying traffic. On my uprights this speed will be closer to 16.
Here is a nice report by an upright rider at a 12hour ultramarathon competition talking about the recumbents competing.
http://dannychew.com/2007Calvins.html

Craig