I notice when my HR is non responsive/low that my workouts are 100% easier (I guess because HR is lower?) and I feel significantly stronger on the bike. That's why I don't let it worry me. Maybe this is just me?
Seriously though after my first VO2 workout in a training block my HR at efforts crashes and doesn't come back up until a rest week.