It's 'running out of oxygen' (due to pushing harder than is sustainable) that causes lactic acid build up...
It sound like you're going too hard in both long and short rides. Reduce the hard efforts to specific intervals >12' at the pace you normally ride the 20m rides and intervals <12' as fast you can without the pace dropping of.