Been riding less due to work hours, so only get out on the weekends. I have noticied that the first few minutes of riding if I put the bike under load, the bottom bracket seems to "grind" a little bit. This goes away after a mile or so.

The bike has 4,000 miles on it and I have never done anything to service the bottom bracket. Should I be dong something about this?

Depends on the BB. If it's sealed, there's not much you can do besides replace it. If it service-able, then dis-assemble, clean and re-grease. Check for play also...another indicator that it's wearing out.

Is it possible that it's one of your pedals?

what BB do you have? Riding in rain and/or dusty conditions?

I would probably re-grease the BB if it's been 4,000 miles. However, like mentioned above, you could have a sealed BB, which isn't serviceable. If that's the case, then I would leave it alone if it's working fine.

if it is sealed, the grinding may just be a bit of dirt working its way along the axle into the BB. You might want to pull the crank out, wipe it clean and regrease the axle before re-inserting. That is probably all you can do.