That's not odd at all, it's the same reason that cushy padded saddles can cause leg numbness, the padding compresses even more soft tissue and nerves than the unpadded versions.

OP may be describing Cyclist Palsy:

7 Ways to Prevent and Treat Cyclist's Palsy | ACTIVE

Grant Peterson, the Rivendell and former Bridgestone guru, believes strongly that if you need padded gloves that what you really need is to raise your handlebars up higher, which is also one of the solutions you'll find on the Active.com linked page.

Your mileage may vary but I found that problem all but went away once I moved from aluminum framed bikes to steel and titanium, even when all the bikes involved had carbon forks.

I can still get left hand numbness and tingling on steel or ti after very long rides without a break, but where I used to get that less than 30 miles into a ride in aluminum, I rarely get that before 85-100 miles on a steel or ti bike and only if I don't take breaks. The theory is that micro vibrations cause the issue as well as pressure on the nerve, and aluminum is the least effective frame material at damping or soaking up those micro vibrations.

And yes, for me it's definitely worse on chipseal and almost never happens on glass smooth asphalt.

If you do get it, my advice is take all the pressure off the affected hand and either ride very casually sitting upright or stop and take a break and walk around.

The longer you let the numbness build up the longer it takes to go away.

Then when you are done with the ride, try raising your handlebars higher. Might need a new stem to do that if you've already cut your steerer down.

And then try out a steel or ti bike if you are riding aluminum.