I'd say do it at your own risk. Just keep in mind that the RR has their own law enforcement, like transit cops, who occasionally patrol RR sections. Eventhough what you have in mind is completely innocent, you have to understand there are a lot of punk kids who dont mind messing with track switches. You know, the same kind of punk kids who throw concrete blocks off of overpasses. If a train derails, and its investigated by the NPSB, there will be a possibility of you being implicated in that, just because you were seen riding your bike along the tracks.
Something to think about!