You can do it... but you would want a bike with a more "normal" frame - like the Formula 1 bikes built for asphalt racing back in the 80s. The stock handlebar had too low a reach but a Wald High Rise Bar solved the problem - and I transformed a Schwinn Qualifier into a serviceable mini velo for not too much money. It rides quite fast on Greenspeed Scorcher tires.

Its more of a fun bike for me - the 20" size being a novelty at this point.