The pitch is shockingly similar. 10-32 = 32 threads per inch. M5x0.8 = 0.8 mm / thread. 1/0.8 = 1.25 threads per mm. 1.25mm * 25.4mm/in = 31.75 threads per inch.
They both use a 60deg angle on the threads.
Pitch diameter of 10-32 is 0.1697". M5x0.8 is 4.480mm = 0.1763".
10-32 UNF-2A (external thread) Major diameter= 0.1891" max - 0.1831" min
M5x0.8 Major diameter female = 5mm = 0.1969"
That gives you about 0.010" overlap, where normaly on a M5 thread you'd have 0.017-0.018" overlap. So, approx. half.
Mind you, except maybe water bottles, the majority of fender and rack mountings are largely subject to radial as opposed to axial forces. So, in general, you should be alright, but in my opinion you'd might as well us a proper M5 bolt if you can.